We discuss the constraints set on galaxy evolution by a variety of data from deep extragalactic surveys performed in the mid-IR and far-IR with the Infrared Space Observatory and with millimetric telescopes at longer wavelengths. These observations indicate extremely high rates of evolution for IR galaxies, exceeding those measured for galaxies at other wavelengths and comparable or larger than the rates observed for quasars. We also match the modelled integrated emission by IR galaxies at any redshifts with the observed spectral intensity of the extragalactic IR background (CIRB), as a further constraint. The multi-wavelength statistics on IR galaxies can be reconciled with each other by assuming for the bulk of the population spectral energy distributions (SED) as typical for starbursts, which we take as an indication that stellar (rather than AGN, see also Fadda et al. cite{fad01}) activity powers IR emission by faint galaxies. According to our model and following the analysis of Elbaz et al. (cite{elb01}), the deep ISO surveys at 15 mu m may have already resolved more than 50% of the bolometric CIRB intensity: the faint ISO 15 mu m source samples, relatively easy to identify in deep optical images (Aussel et al. cite{aus99}), can then allow to investigate the origin of the CIRB background. From our fits to the observed optical-IR SEDs, these objects appear to mostly involve massive galaxies hosting luminous starbursts (SFR ~ 100 Msun/yr). The evolutionary scheme we infer from these data considers a bimodal star formation (SF), including a phase of long-lived quiescent SF, and enhanced SF taking place during transient events recurrently triggered by interactions and merging. We interpret the strong observed evolution as an increase with z of the rate of interactions between galaxies (density evolution) and an increase of their IR luminosity due to the more abundant fuel available in the past (luminosity evolution): both factors enhance the probability to detect a galaxy during the ``active" phase at higher z. Very schematically, we associate the origin of the bulk of the optical/NIR background to the quiescent evolution, while the CIRB is interpreted as mostly due the dusty starburst phase. The latter possibly leads to the formation of galaxy spheroids, when the dynamical events triggering the starburst re-distribute already present stellar populations. The large energy contents in the CIRB and optical backgrounds are not easily explained, considering the moderate efficiency of energy generation by stars: a top-heavy stellar IMF associated with the starburst phase (and compared with a more standard IMF during the quiescent SF) would alleviate the problem. The evolution of the IR emissivity of galaxies from the present time to z ~ 1 is so strong that the combined set of constraints by the observed z-distributions and the CIRB spectrum impose it to turn-over at z>1: scenarios in which a dominant fraction of stellar formation occurs at very high-z are not supported by our analysis.