We present the largest-scale comparison to date between observed extragalactic X-ray binary (XRB) populations and theoretical models of their production. We construct observational X-ray luminosity functions (oXLFs) using Chandra observations of 12 late-type galaxies from the Spitzer Infrared Nearby Galaxy Survey. For each galaxy, we obtain theoretical XLFs (tXLFs) by combining XRB synthetic models, constructed with the population synthesis code StarTrack, with observational star formation histories (SFHs). We identify highest-likelihood models both for individual galaxies and globally, averaged over the full galaxy sample. Individual tXLFs successfully reproduce about half of the oXLFs, but for some galaxies we are unable to find underlying source populations, indicating that galaxy SFHs and metallicities are not well matched and/or that XRB modeling requires calibration on larger observational samples. Given these limitations, we find that the best models are consistent with a product of common envelope ejection efficiency and central donor concentration =~0.1, and a 50% uniform-50% "twins" initial mass-ratio distribution. We present and discuss constituent subpopulations of tXLFs according to donor, accretor, and stellar population characteristics. The galaxy-wide X-ray luminosity due to low-mass and high-mass XRBs, estimated via our best global model tXLF, follows the general trend expected from the L_X_-star formation rate and L_X_-stellar mass relations of Lehmer et al. Our best models are also in agreement with modeling of the evolution of both XRBs over cosmic time and of the galaxy X-ray luminosity with redshift.