We construct rest-frame luminosity functions (LFs) at 3.6, 4.5, 5.8, 8 and 24 μm over the redshift range 0 < z < 2 for galaxies and 0 < z < 4 for optical quasi-stellar objects (QSOs), using optical and infrared (IR) data from the Spitzer Wide-area Infrared Extragalactic (SWIRE) Survey. The 3.6- and 4.5-μm galaxy LFs show evidence for moderate positive luminosity evolution up to z ~ 1.5, consistent with the passive ageing of evolved stellar populations. Their comoving luminosity density was found to evolve passively, gradually increasing out to z ~ 0.5-1 but flattening, or even declining, at higher redshift. Conversely, the 24-μm galaxy LF, which is more sensitive to obscured star formation and/or active galactic nuclei (AGN) activity, undergoes strong positive evolution, with the derived IR energy density and star formation rate (SFR) density ~ (1 + z)γ with γ = 4.5+0.7-0.6 and the majority of this evolution occurring since z ~ 1. Optical QSOs, however, show positive luminosity evolution in all bands, out to the highest redshifts (3 < z < 4). Modelling as L* ~ (1 + z)γ gave γ = 1.3+0.1-0.1 at 3.6μm,γ = 1.0+0.1-0.1 at 4.5μm and stronger evolution at the longer wavelengths (5.8, 8 and 24μm), of γ ~ 3. Comparison of the galaxy LFs to predictions from a semi-analytic model based on cold dark matter (CDM) indicates that an initial mass function (IMF) skewed towards higher mass star formation in bursts compared to locally be preferred. As a result, the currently inferred massive SFRs in distant submm sources may require substantial downwards revision.