The Chandra Deep Field-South and North surveys (CDFs) provide unique windows into the cosmic history of X-ray emission from normal (nonactive) galaxies. Scaling relations of normal-galaxy X-ray luminosity (L_X_) with star formation rate (SFR) and stellar mass (M_*_) have been used to show that the formation rates of low-mass and high-mass X-ray binaries (LMXBs and HMXBs, respectively) evolve with redshift across z~0-2 following L_HMXB_/SFR{propto}(1+z) and L_LMXB_/M*{propto}(1+z)^2-3^. However, these measurements alone do not directly reveal the physical mechanisms behind the redshift evolution of X-ray binaries (XRBs). We derive star formation histories for a sample of 344 normal galaxies in the CDFs, using spectral energy distribution (SED) fitting of FUV-to-FIR photometric data, and construct a self-consistent, age-dependent model of the X-ray emission from the galaxies. Our model quantifies how X-ray emission from hot gas and XRB populations vary as functions of host stellar-population age. We find that (1) the ratio L_X_/M_*_ declines by a factor of ~1000 from 0 to 10Gyr and (2) the X-ray SED becomes harder with increasing age, consistent with a scenario in which the hot gas contribution to the X-ray SED declines quickly for ages above 10Myr. When dividing our sample into subsets based on metallicity, we find some indication that L_X_/M_*_ is elevated for low-metallicity galaxies, consistent with recent studies of X-ray scaling relations. However, additional statistical constraints are required to quantify both the age and metallicity dependence of X-ray emission from star-forming galaxies.