The presence of giant planets influences potentially habitable worlds in numerous ways. Massive celestial neighbors can facilitate the formation of planetary cores and modify the influx of asteroids and comets toward Earth analogs later on. Furthermore, giant planets can indirectly change the climate of terrestrial worlds by gravitationally altering their orbits. Investigating 147 well-characterized exoplanetary systems known to date that host a main-sequence star and a giant planet, we show that the presence of "giant neighbors" can reduce a terrestrial planet's chances to remain habitable, even if both planets have stable orbits. In a small fraction of systems, however, giant planets slightly increase the extent of habitable zones provided that the terrestrial world has a high climate inertia. In providing constraints on where giant planets cease to affect the habitable zone size in a detrimental fashion, we identify prime targets in the search for habitable worlds.