The complex structure of gas, metals, and dust in the interstellar and circumgalactic medium (ISM and CGM, respectively) in star-forming galaxies can be probed by Ly{alpha} emission and absorption, low-ionization interstellar (LIS) metal absorption, and dust reddening E(B-V). We present a statistical analysis of the mutual correlations among Ly{alpha} equivalent width (EWLy{alpha}), LIS equivalent width (EWLIS), and E(B-V) in a sample of 157 star-forming galaxies at z~2.3. With measurements obtained from individual deep rest-UV spectra and spectral energy distribution modeling, we find that the tightest correlation exists between EWLIS and E(B-V), although correlations among all three parameters are statistically significant. These results signal a direct connection between dust and metal-enriched HI gas and that they are likely cospatial. By comparing our results with the predictions of different ISM/CGM models, we favor a dusty ISM/CGM model where dust resides in HI gas clumps and Ly{alpha} photons escape through the low HI covering fraction/column density intraclump medium. Finally, we investigate the factors that potentially contribute to the intrinsic scatter in the correlations studied in this work, including metallicity, outflow kinematics, Ly{alpha} production efficiency, and slit loss. Specifically, we find evidence that scatter in the relationship between EWLy{alpha} and E(B-V) reflects the variation in the metal-to-H i covering fraction ratio as a function of metallicity and the effects of outflows on the porosity of the ISM/CGM. Future simulations incorporating star formation feedback and the radiative transfer of Ly{alpha} photons will provide key constraints on the spatial distributions of neutral hydrogen gas and dust in the ISM/CGM structure.