We combine samples of nearby galaxies with Herschel photometry selected on their dust, metal, HI and stellar mass content, and compare these to chemical evolution models in order to discriminate between different dust sources. In a companion paper, we used an HI-selected sample of nearby galaxies to reveal a subsample of very gas-rich (gas fraction >80 per cent) sources with dust masses significantly below predictions from simple chemical evolution models, and well below M_d_/M_*_ and M_d_/M_gas_ scaling relations seen in dust and stellar-selected samples of local galaxies. We use a chemical evolution model to explain these dust-poor, but gas-rich, sources as well as the observed star formation rates (SFRs) and dust-to-gas ratios. We find that (i) a delayed star formation history is required to model the observed SFRs; (ii) inflows and outflows are required to model the observed metallicities at low gas fractions; (iii) a reduced contribution of dust from supernovae (SNe) is needed to explain the dust-poor sources with high gas fractions. These dust-poor, low stellar mass galaxies require a typical core-collapse SN to produce 0.01-0.16M_{sun}_ of dust. To match the observed dust masses at lower gas fractions, significant grain growth is required to counteract the reduced contribution from dust in SNe and dust destruction from SN shocks. These findings are statistically robust, though due to intrinsic scatter it is not always possible to find one single model that successfully describes all the data. We also show that the dust-to-metal ratio decreases towards lower metallicity.