Between 1997 and 2004 several observing runs were conducted mainly with the CTIO 0.9m to image ICRF counterparts (mostly QSOs) in order to determine accurate optical positions. Contemporary to these deep CCD images the same fields were observed with the USNO astrograph in the same bandpass. These provide accurate positions on the Hipparcos/Tycho-2 system for stars in the 10 to 16 magnitude range used as reference stars for the deep CCD imaging data. Here we present final optical position results of 413 sources based on astrograph data reductions using UCAC2 and UCAC4 type reference stars. These optical positions are compared to radio VLBI positions. The current optical system is not perfectly aligned to the ICRF radio system with rigid body rotation angles of 3 to 5 mas (3{sigma}) found between them for all 3 axes. Furthermore, statistically, the optical-radio position differences are found to exceed the total, combined, known errors in the observations. Systematic errors in the optical reference stars as well as physical offsets between the centers of optical and radio emissions are both identified as likely causes. A detrimental, astrophysical, random noise (DARN) component is postulated to be on about the 10mas level. If confirmed by future observations, this could severely limit the Gaia to ICRF reference frame alignment accuracy to an error of about 0.5mas per coordinate axis with the current number of sources envisioned to provide the link. A list of 36 ICRF sources without the detection of an optical counterpart to a limiting magnitude of about R=22 is provided as well.