Planet formation is generally described in terms of a system containing the host star and a protoplanetary disk, of which the internal properties (for example, mass and metallicity) determine the properties of the resulting planetary system. However, (proto)planetary systems are predicted and observed to be affected by the spatially clustered stellar formation environment, through either dynamical star-star interactions or external photoevaporation by nearby massive stars. It is challenging to quantify how the architecture of planetary sysems is affected by these environmental processes, because stellar groups spatially disperse within less than a billion years, well below the ages of most known exoplanets. Here we identify old, co-moving stellar groups around exoplanet host stars in the astrometric data from the Gaia satellite and demonstrate that the architecture of planetary systems exhibits a strong dependence on local stellar clustering in position-velocity phase space. After controlling for host stellar age, mass, metallicity and distance from the star, we obtain highly significant differences (with p values of 10^-5^ to 10^-2^) in planetary system properties between phase space overdensities (composed of a greater number of co-moving stars than unstructured space) and the field. The median semi-major axis and orbital period of planets in phase space overdensities are 0.087 astronomical units and 9.6 days, respectively, compared to 0.81 astronomical units and 154 days, respectively, for planets around field stars. 'Hot Jupiters' (massive, short-period exoplanets) predominantly exist in stellar phase space overdensities, strongly suggesting that their extreme orbits originate from environmental perturbations rather than internal migration or planet-planet scattering. Our findings reveal that stellar clustering is a key factor setting the architectures of planetary systems.