The extragalactic distance scale builds on the Cepheid period-luminosity (PL) relation. Decades of work have not yet convincingly established the sensitivity of the PL relation to metallicity. This currently prevents a determination of the Hubble constant accurate to 1% from the classical Cepheid-SN Ia method. In this paper we carry out a strictly differential comparison of the absolute PL relations obeyed by classical Cepheids in the Milky Way (MW), LMC, and SMC galaxies. Taking advantage of the substantial metallicity difference among the Cepheid populations in these three galaxies, we want to establish a possible systematic trend of the PL relation absolute zero point as a function of metallicity, and to determine the size of such an effect in the optical and near-infrared photometric bands. We used a IRSB Baade-Wesselink-type method to determine individual distances to the Cepheids in our samples in the MW, LMC, and SMC. For our analysis, we used a greatly enhanced sample of Cepheids in the SMC (31 stars) compared to the small sample (5 stars) available in our previous work. We used the distances to determine absolute Cepheid PL relations in the optical and near-infrared bands in each of the three galaxies. Our distance analysis of 31 SMC Cepheids with periods of 4-69 days yields tight PL relations in all studied bands, with slopes consistent with the corresponding LMC and MW relations. Adopting the very accurately determined LMC slopes for the optical and near-infrared bands, we determine the zero point offsets between the corresponding absolute PL relations in the three galaxies. We find that in all bands the metal-poor SMC Cepheids are intrinsically fainter than their more metal-rich counterparts in the LMC and MW. In the K band the metallicity effect is -0.23+/-0.06mag/dex, while in the V,(V-I) Wesenheit index it is slightly stronger, -0.34+/-0.06mag/dex. We find suggestive evidence that the metallicity sensitivity of the PL relation might be nonlinear, being small in the range between solar and LMC Cepheid metallicity, and becoming steeper towards the lower-metallicity regime.