Recent results on the non-universality of the Initial Mass Function (IMF) have shown strong evidence of IMF variations with galaxy velocity dispersion, with a corresponding impact on other stellar population parameters, line indices and colours. Using a set of stellar population models with varying IMF slopes, we provide additional caveats on the assumption of a universal IMF. This study shows that the derived star formation histories of early-type galaxies vary significantly with the IMF slope. For instance, a steepening in the slope of a single power-law IMF decreases substantially, by a factor of up to 4, the contribution of the old stellar populations to the total light/mass. This trend is milder for a segmented-like IMF shape, where the contribution of the very low mass stars is decreased. It is also shown that, by tuning each IMF slope to its prescribed value according to each galaxy velocity dispersion, a sample of early-type galaxies covering a range of masses yield comparable star formation histories. On the one hand, a small contribution from relatively young stellar populations appears in the star formation histories of most massive elliptical galaxies when adopting a steep IMF. In addition, we find that low-mass early-type galaxies that look like genuinely young objects with a standard IMF (i.e. `baby elliptical galaxies') turn out to be older when a slightly flatter IMF is employed. In summary, the use of a non-universal IMF, tuned according to the velocity dispersion of the galaxy, seems to provide more consistent results.