Hilda asteroids and Jupiter Trojans are two low-albedo (p_v_~0.07) populations for which the Nice model predicts an origin in the primordial Kuiper Belt region. However, recent surveys by WISE and the Spitzer Space Telescope (SST) have revealed that ~2% of these objects possess high albedos (p_v_>=0.15), which might indicate interlopers --that is, objects not formed in the Kuiper Belt-- among these two populations. Here, we report spectroscopic observations in the visible and/or near-infrared spectral ranges of twelve high-albedo (p_v_>0.15) Hilda asteroids and Jupiter Trojans. These twelve objects have spectral properties similar to those of the low-albedo population, which suggests a similar composition and hence a similar origin for low- and high-albedo Hilda asteroids and Jupiter Trojans. We therefore propose that most high albedos probably result from statistical bias or uncertainties that affect the WISE and SST measurements. However, some of the high albedos may be true and the outcome of some collision-induced resurfacing by a brighter material that could include water ice. Future work should attempt to investigate the nature of this supposedly bright material. The lack of interlopers in our sample allows us to set an upper limit of 0.4% at a confidence level of 99.7% on the abundance of interlopers with unexpected taxonomic classes (e.g., A-, S-, V-type asteroids) among these two populations.