Characterizing the relation between stellar photometric variability and radial velocity (RV) jitter can help us to better understand the physics behind these phenomena. The current and upcoming high precision photometric surveys such as TESS, CHEOPS, and PLATO will provide the community with thousands of new exoplanet candidates. As a consequence, the presence of such a correlation is crucial in selecting the targets with the lowest RV jitter for efficient RV follow-up of exoplanetary candidates. Studies of this type are also crucial to design optimized observational strategies to mitigate RV jitter when searching for Earth-mass exoplanets. Aims. Our goal is to assess the correlation between high-precision photometric variability measurements and high-precision RV jitter over different time scales. We analyze 171 G, K, and M stars with available TESS high precision photometric time-series and HARPS precise RVs. We derived the stellar parameters for the stars in our sample and measured the RV jitter and photometric variability. We also estimated chromospheric CaII H & K activity indicator log(R'_HK_), vsini, and the stellar rotational period. Finally, we evaluate how different stellar parameters and an RV sampling subset can have an impact on the potential correlations. We find a varying correlation between the photometric variability and RV jitter as function of time intervals between the TESS photometric observation and HARPS RV. As the time intervals of the observations considered for the analysis increases, the correlation value and significance becomes smaller and weaker, to the point that it becomes negligible. We also find that for stars with a photometric variability above 6.5 ppt the correlation is significantly stronger. We show that such a result can be due to the transition between the spot-dominated and the faculae-dominated regime. We quantified the correlations and updated the relationship between chromospheric CaII H & K activity indicator log(R'_HK_) and RV jitter.