Lyman alpha emitters (LAEs) and Lyman break galaxies (LBGs) represent the most common groups of star-forming galaxies at high z, and the differences between their inherent stellar populations (SPs) are a key factor in understanding early galaxy formation and evolution. We have run a set of SP burst-like models for a sample of 1558 sources at 3.4 < z < 6.8 from the Survey for High-z Absorption Red and Dead Sources (SHARDS) over the GOODS-N field. This work focuses on the differences between the three different observational subfamilies of our sample: LAE-LBGs, no-Ly α LBGs, and pure LAEs. Single and double SP synthetic spectra were used to model the spectral energy distributions, adopting a Bayesian information criterion to analyze under which situations a second SP is required. We find that the sources are well modelled using a single SP in $\sim 79{{\ \rm per\ cent}}$ of the cases. The best models suggest that pure LAEs are typically young low-mass galaxies ( $t\sim 26^{+41}_{-25}$ Myr; $M_{\mathrm{star}}\sim 5.6^{+12.0}_{-5.5}\times 10^{8}\ M_{\odot }$ ), undergoing one of their first bursts of star formation. On the other hand, no-Ly α LBGs require older SPs (t ∼ 71 ± 12 Myr), and they are substantially more massive (Mstar ∼ 3.5 ± 1.1 × 109 M☉). LAE-LBGs appear as the subgroup that more frequently needs the addition of a second SP, representing an old and massive galaxy caught in a strong recent star-forming episode. The relative number of sources found from each subfamily at each z supports an evolutionary scenario from pure LAEs and single SP LAE-LBGs to more massive LBGs. Stellar mass functions are also derived, finding an increase of M* with cosmic time and a possible steepening of the low-mass slope from z ∼ 6 to z ∼ 5 with no significant change to z ∼ 4. Additionally, we have derived the SFR-Mstar relation, finding an $\mathrm{SFR}\propto M_{\mathrm{star}}^{\beta }$ behaviour with negligible evolution from z ∼ 4 to z ∼ 6.