We investigate two-dimensional image decomposition of nearby, morphologically selected early-type galaxies (ETGs). We are motivated by recent observational evidence of significant size growth of quiescent galaxies and theoretical development advocating a two-phase formation scenario for ETGs. We find that a significant fraction of nearby ETGs show changes in isophotal shape that require multi-component models. The characteristic sizes of the inner and outer component are ~3 and ~15kpc. The inner component lies on the mass-size relation of ETGs at z~0.25-0.75, while the outer component tends to be more elliptical and hints at a stochastic buildup process. We find real physical differences between single- and double-component ETGs, with double-component galaxies being younger and more metal-rich. The fraction of double-component ETGs increases with increasing {sigma} and decreases in denser environments. We hypothesize that double-component systems were able to accrete gas and small galaxies until later times, boosting their central densities, building up their outer parts, and lowering their typical central ages. In contrast, the oldest galaxies, perhaps due to residing in richer environments, have no remaining hints of their last accretion episode.