High-redshift galaxies permit the study of the formation and evolution of X-ray binary (XRB) populations on cosmological timescales, probing a wide range of metallicities and star formation rates (SFRs). In this paper, we present results from a large-scale population synthesis study that models the XRB populations from the first galaxies of the universe until today. We use as input to our modeling the Millennium II cosmological simulation (Boylan-Kolchin et al. 2009MNRAS.398.1150B) and the updated semi-analytic galaxy catalog by Guo et al. (2011MNRAS.413..101G) to self-consistently account for the star formation history and metallicity evolution of the universe. Our modeling, which is constrained by the observed X-ray properties of local galaxies, gives predictions about the global scaling of emission from XRB populations with properties such as SFR and stellar mass, and the evolution of these relations with redshift. Our simulations show that the X-ray luminosity density (X-ray luminosity per unit volume) from XRBs in our universe today is dominated by low-mass XRBs, and it is only at z>~2.5 that high-mass XRBs become dominant. We also find that there is a delay of ~1.1Gyr between the peak of X-ray emissivity from low-mass XRBs (at z~2.1) and the peak of SFR density (at z~3.1). The peak of the X-ray luminosity from high-mass XRBs (at z~3.9) happens ~0.8Gyr before the peak of the SFR density, which is due to the metallicity evolution of the universe.