The stellar initial mass function (IMF) is central to our interpretation of astronomical observables and to our understanding of most baryonic processes within galaxies. The universality of the IMF, suggested by observations in our own Milky Way, has been thoroughly revisited due to the apparent excess of low-mass stars in the central regions of massive quiescent galaxies. As part of the efforts within the Fornax 3D project, we aim to characterize the two-dimensional IMF variations in a sample of 23 quiescent galaxies within the Fornax cluster. For each galaxy in the sample, we measured the mean age, metallicity, [Mg/Fe], and IMF slope maps from spatially resolved integrated spectra. The IMF maps show a variety of behaviors and internal substructures, roughly following metallicity variations. However, metallicity alone is not able to fully explain the complexity exhibited by the IMF maps. In particular, for relatively metal-poor stellar populations ([M/H] ≲ −0.1), the slope of the IMF seems to depend on the (specific) star formation rate at which stars were formed. Moreover, metallicity maps have systematically higher ellipticities than IMF slope ones. At the same time, both metallicity and IMF slope maps have at the same time higher ellipticities than the stellar light distribution in our sample of galaxies. In addition we find that, regardless of the stellar mass, every galaxy in our sample shows a positive radial [Mg/Fe] gradient. This results in a strong [Fe/H]-[Mg/Fe] relation, similar to what is observed in nearby, resolved galaxies. Since the formation history and chemical enrichment of galaxies are causally driven by changes in the IMF, our findings call for a physically motivated interpretation of stellar population measurements based on integrated spectra that take into account any possible time evolution of the stellar populations.