Lyman alpha emitters (LAEs) and Lyman break galaxies (LBGs) represent the most common groups of star-forming galaxies at high z, and the differences between their inherent stellar populations (SPs) are a key factor in understanding early galaxy formation and evolution. We have run a set of SP burst-like models for a sample of 1558 sources at 3.4<z<6.8 from the Survey for High-z Absorption Red and Dead Sources (SHARDS) over the GOODS-N field. This work focuses on the differences between the three different observational subfamilies of our sample: LAE-LBGs, no-Ly{alpha} LBGs, and pure LAEs. Single and double SP synthetic spectra were used to model the spectral energy distributions, adopting a Bayesian information criterion to analyze under which situations a second SP is required. We find that the sources are well modelled using a single SP in ~79 per cent of the cases. The best models suggest that pure LAEs are typically young low-mass galaxies (t~26^+41^_-25_Myr; M_star_~5.6^+12.0^_-5.5_x10^8^M_{sun}_), undergoing one of their first bursts of star formation. On the other hand, no-Ly{alpha} LBGs require older SPs (t~71+/-12Myr), and they are substantially more massive (M_star_~3.5+/-1.1x10^9^M_{sun}_). LAE-LBGs appear as the subgroup that more frequently needs the addition of a second SP, representing an old and massive galaxy caught in a strong recent star-forming episode. The relative number of sources found from each subfamily at each z supports an evolutionary scenario from pure LAEs and single SP LAE-LBGs to more massive LBGs. Stellar mass functions are also derived, finding an increase of M_*_ with cosmic time and a possible steepening of the low-mass slope from z~6 to z~5 with no significant change to z~4. Additionally, we have derived the SFR-M_star_ relation, finding an SFR{prop.to}M_star_^{beta}^ behaviour with negligible evolution from z~4 to z~6.