The detection of Earth-like planets with the radial velocity (RV) method is extremely challenging today due to the presence of non-Doppler signatures such as stellar activity and instrumental signals that mimic and hide the signals of exoplanets. In a previous paper, we presented the YARARA pipeline, which implements corrections for telluric absorption, stellar activity, and instrumental systematics at the spectral level, and then it extracts line-by-line (LBL) RVs with a significantly better precision than standard pipelines. In this paper, we demonstrate that further gains in RV precision can be achieved by performing principal component analysis (PCA) decomposition on the LBL RVs. The mean-insensitive nature of PCA means that it is unaffected by true Doppler shifts, and thus can be used to isolate and correct nuisance signals other than planets. We analysed the data of 20 intensively observed HARPS targets by applying our PCA approach on the LBL RVs obtained by YARARA. The first principal components show similarities across most of the stars and correspond to newly identified instrumental systematics for which we can now correct. For several targets, this results in an unprecedented RV root-mean-square of around 90cm/s over the full lifetime of HARPS. We used the corrected RVs to confirm a previously published 120-day signal around 61 Vir, and to detect a super-Earth candidate (K~60+/-6cm/s, msini=6.6+/-0.7M+{Earth}_) around the G6V star HD 20794, which spends part of its 600-day orbit within the habitable zone of the host star. This study highlights the potential of LBL PCA to identify and correct hitherto unknown, long-term instrumental effects and thereby extend the sensitivity of existing and future instruments towards the Earth analogue regime.