We present a study of the correlation between the direction of the symmetry axis of the circumstellar material around intermediate mass young stellar objects and that of the interstellar magnetic field. We use CCD polarimetric data on 100 Herbig Ae/Be stars. A large number of them show intrinsic polarization, which indicates that their circumstellar envelopes are not spherical. The interstellar magnetic field direction is estimated from the polarization of field stars. There is an alignment between the position angle of the Herbig Ae/Be star polarization and that of the field stars for the most polarized objects. This may be an evidence that the ambient interstellar magnetic field plays a role in shaping the circumstellar material around young stars of intermediate mass and/or in defining their angular momentum axis.