Ever since the very first photometric studies of Centaurs and Kuiper belt objects (KBOs) their visible color distribution has been controversial. This controversy has triggered a prolific debate on the origin of the surface colors of these distant icy objects of the solar system. Two scenarios have been proposed to interpret and explain the large variability of colors, hence surface composition. Are the colors mainly primordial and directly related to the formation region, or are they the result of surface evolution processes? To date, no mechanism has been found that successfully explains why Centaurs, which are escapees from the Kuiper belt, exhibit two distinct color groups, whereas KBOs do not. We re-address this issue using a carefully compiled set of B-R colors and H_R{alpha}_ magnitudes (as proxy for size) for 253 objects, including data for 10 new small objects. We find that the bimodal color distribution of Centaurs is a size-related phenomenon, common to both Centaurs and small KBOs, i.e. independent of dynamical classification. Furthermore, we find that large KBOs also have a bimodal distribution of surface colors, albeit distinct from the small objects and strongly dependent on the `Haumea collisional family' objects. When plotted in B-R, H_R{alpha}_ space, the colors of Centaurs and KBOs display a peculiar N shape.