We characterize the optical counterparts to the compact X-ray source population within the nearby spiral galaxy M81 using multiband Hubble Space Telescope (HST) imaging data. By comparing the optical luminosities and colors measured for candidate donor stars and host clusters to stellar and cluster evolutionary models, respectively, we estimate the likely masses and upper age limits of the field and cluster X-ray binaries. We identify 15 low-mass X-ray binaries (i.e., donor star mass <~3M_{sun}_) within ancient globular clusters, as well as 42 candidate high-mass X-ray binaries (i.e., donor star mass >~8M_{sun}_). To estimate the likelihood of misclassifications, we inject 4000 artificial sources into the HST mosaic image and conclude that our classifications of globular clusters and high-mass X-ray binaries are reliable at the >90% level. We find that globular clusters that host X-ray binaries are on average more massive and more compact than globular clusters that do not. However, there is no apparent correlation between the X-ray brightness of the clusters and their masses or densities, nor are X-ray binary hosts more X-ray luminous than the general field population of low-mass X-ray binaries. This work represents one of the first in-depth analyses of the population of X-ray binaries within globular clusters in a spiral galaxy.