Numerous studies have reported significant displacements in the coordinates of active galactic nuclei (AGN) between measurements using the Very Long Baseline Interferometry (VLBI) technique and those obtained by the Gaia space observatory. There is consensus that these discrepancies do indeed manifest astrometrically resolved sub-components of AGN rather than random measurement noise. Among other evidence, it has been reported that AGN with VLBI-to-Gaia displacements (VGD) pointing downstream their parsec-scale radio jets exhibit higher optical polarization compared to sources with the opposite (upstream) VGD orientation.We aim to verify the previously reported connection between optical polarization and VGD-jet angle using a larger dataset of polarimetric measurements and updated Gaia DR3 positions. We also seek further evidence supporting the disc-jet dichotomy as an explanation of such a connection by using mm-wave polarization and multi-band optical polarization measurements. We performed optical polarimetric observations of 152 AGN using three telescopes. These data are complemented by other publicly available polarimetric measurements of AGN. We cross-matched public astrometric data from VLBI and Gaia experiments, obtain corresponding positional displacements, and combined this catalog with the polarimetric and jet directions data. AGN with downstream VGD are confirmed to have significantly higher optical fractional polarization than the upstream sample. At the same time, the mm-wavelength polarization of the two samples shows very similar distributions. Our results support the hypothesis that the VLBI-to-Gaia displacements pointing down the radio jet are likely caused by a component in the jet emitting highly polarized synchrotron radiation and dominating in the overall optical emission. The upstream-oriented VGD are likely to be produced by the low-polarized emission of the central engine sub-components, which dominate in the optical.