We present new information on galaxies in the vicinity of luminous radio galaxies (RGs) and quasars at z~4, 5, and 6. These fields were previously found to contain overdensities of Lyman Break Galaxies (LBGs) or spectroscopic Ly{alpha} emitters (LAEs), which were interpreted as evidence for clusters-in-formation ("protoclusters"). We use Hubble Space Telescope and Spitzer data to infer stellar masses from stellar synthesis models calibrated against the Millennium Run simulations, and contrast our results with large samples of LBGs in more average environments as probed by the Great Observatories Origins Deep Survey (GOODS).