Timing observations of rapidly rotating neutron stars revealed a great number of glitches, observed from both canonical radio pulsars and magnetars. Among them, 76 glitches have shown exponential relaxation(s) with characteristic decay times ranging from several days to a few months, followed by a more gradual recovery. Glitches displaying exponential relaxation with single or multiple decay time constants are analysed in terms of a model based on the interaction of the vortex lines with the toroidal arrangement of flux tubes in the outer core of the neutron star. Model results agree with the observed time-scales in general. Thus, the glitch phenomenon can be used to deduce valuable information about neutron star structure, in particular on the interior magnetic field configuration which is unaccessible from surface observations. One immediate conclusion is that the magnetar glitch data are best explained with a much cooler core and therefore require that direct Urca-type fast-cooling mechanisms should be effective for magnetars.