Galaxy interactions distort the distribution of baryonic matter and can affect star formation. The nearby Magellanic Clouds are a prime example of an ongoing galaxy interaction process. Here we use the intermediate-age (~1-10Gyr) red clump stars to map the three-dimensional structure of the Small Magellanic Cloud (SMC) and interpret it within the context of its history of interaction with the Large Magellanic Cloud (LMC) and the Milky Way. Red clump stars are selected from near-infrared colour--magnitude diagrams based on data from the VISTA survey of the Magellanic Clouds. Interstellar reddening is measured and removed, and the corrected brightness is converted to a distance, on a star-by-star basis. A flat plane fitted to the spatial distribution of red clump stars has an inclination i=35{deg}-48{deg} and position angle PA=170{deg}-186{deg}. However, significant deviations from this plane are seen, especially in the periphery and on the eastern side of the SMC. In the latter part, two distinct populations are present, separated in distance by as much as 10kpc. Distant red clump stars are seen in the North of the SMC, and possibly also in the far West; these might be associated with the predicted `Counter-Bridge'. We also present a dust reddening map, which shows that dust generally traces stellar mass. The structure of the intermediate-age stellar component of the SMC bears the imprints of strong interaction with the LMC a few Gyr ago, which cannot be purely tidal but must have involved ram pressure stripping.