We investigate a sample of 3413 International Celestial Reference Frame (ICRF3) extragalactic radio-loud sources with accurate positions determined by very long baseline interferometry in the S/X band, mostly active galactic nuclei and quasars, which are cross-matched with optical sources in the second Gaia data release (Gaia DR2). The main goal of this study is to determine a core sample of astrometric objects that define the mutual orientation of the two fundamental reference frames, the Gaia (optical) and the ICRF3 (radio) frames. The distribution of normalized offsets between the VLBI sources and their optical counterparts is non-Rayleigh, with a deficit around the modal value and a tail extending beyond the 3{sigma} confidence level. A few filters are applied to the sample in order to discard double cross-matches, confusion sources, and Gaia astrometric solutions of doubtful quality. Panoramic Survey Telescope and Rapid Response System and Dark Energy Survey stacked multicolor images are used to further deselect objects that are less suitable for precision astrometry, such as extended galaxies, double and multiple sources, and obvious misidentifications. After this cleaning, 2643 quasars remain, of which 20% still have normalized offset magnitudes exceeding 3, or a 99% confidence level. We publish a list of 2119 radio-loud quasars of prime astrometric quality. The observed dependence of binned median offset on redshift shows the expected decline at small redshifts, but also an unexpected rise at z~1.6, which may be attributed to the emergence of the CIV emission line in the Gaia's G band. The Gaia DR2 parallax zero-point is found to be color-dependent, suggesting an uncorrected instrumental calibration effect.