We provide constraints on the AGN contribution to the mid-IR extragalactic background light from a correlation analysis of deep X-ray and mid-IR observations in two regions centred on the Lockman Hole and Hubble Deep Field North (HDF-N). The Lockman region, of more than 200 square arcminutes, was observed by ISOCAM and XMM-Newton to a depth of 0.3 mJy at 15 mu m (resolving more than 30% of the mid-IR background). In the same area XMM-Newton reached flux limits of 1.4 x 10-15 erg cm-2 s-1 in the 2-10 keV energy band and 2.4 x 10-15 erg cm-2 s-1 in the 5-10 keV energy band, resolving about 80% of the 2-10 keV and 60% of the 5-10 keV backgrounds (the deepest observation in this hard band to date). Among the 76 galaxies detected by XMM-Newton, 24 show mid-IR emission, but the relative percentage of X-ray sources with mid-IR counterparts increases with the band energy: from 30% of the 0.5-2 keV sources up to 63% of the 5-10 keV sources. In contrast, only a small fraction of the mid-IR sources (around 10%) show X-ray emission within the sensitivity limits of XMM-Newton observations. The region centred on the HDF-N has been observed by ISOCAM (24 square arcminutes) to a depth of 0.05 mJy (more than 60% of the mid-IR background is resolved at this limit) and covered with a 1 Msec exposure by Chandra. In this case, 25% of the mid-IR sources are detected in the X-ray, while 30-40% of the X-ray sources show mid-IR emission. Under the assumption that all XMM sources except stars or galaxy clusters are AGN-dominated, AGNs contribute (15+/-5)% of the total mid-IR flux in the Lockman Hole. For the HDF-N we have assumed that AGN-dominated sources are luminous X-ray sources and sources with SEDs from radio to X-ray wavelengths typical of local AGNs, in which case we find that (18+/-7)% of the mid-IR flux are due to AGN emission. If we put together all the existing information from the deepest HDF-N data to the bright large-area sample in the ELAIS S1 region observed with BeppoSAX (for a total of 50 X-ray-mid-IR matched sources) using the median mid-IR to X-ray spectral indices as a function of the X-ray flux, we find an AGN contribution to the 15 mu m background of (17+/- 2)%. Since the IR spectra of typical AGNs peak around 20 mu m while starburst spectra peak at significantly longer infrared wavelengths, this figure may be considered as an upper limit to the AGN contribution to the CIRB energy density. We conclude that the population of IR luminous galaxies detected in the ISOCAM deep surveys, and the CIRB sources themselves, are mainly constituted by dust-obscured starbursts. Partially based on observations with ISO and XMM, ESA projects with instruments funded by ESA Member States with the participation of ISAS and NASA, and on observations with the Canada-France-Hawaii Telescope at Mauna Kea, Hawaii.