We exploit deep combined observations with Spitzer and Chandra of the Spitzer Wide-Area Infrared Extragalactic Survey (SWIRE) in the ELAIS N1 region to investigate the nature of the faint X-ray and IR sources in common, to identify active galactic nucleus (AGN)/starburst diagnostics, and to study the sources of the X-ray and IR cosmic backgrounds (XRB and CIRB). In the 17'×17' area of the Chandra ACIS-I image there are approximately 3400 SWIRE near-IR sources with 4 σ detections in at least two Infrared Array Camera (IRAC) bands and 988 sources detected at 24 μm with the Multiband Imaging Photometer (MIPS) brighter than S24~=0.1 mJy. Of these, 102 IRAC and 59 MIPS sources have Chandra counterparts, out of a total of 122 X-ray sources present in the area with S0.5-8keV>10-15 ergs cm-2 s-1. We have constructed spectral energy distributions (SEDs) for each source using data from the four IRAC wavebands, Chandra fluxes in the hard (2-8 keV) and soft (0.5-2 keV) X-rays, and optical follow-up data in the wavebands U, g', r', i', Z, and H. We fit a number of spectral templates to the SEDs at optical and IR wavelengths to determine photometric redshifts and spectral categories and also make use of diagnostics based on the X-ray luminosities, hardness ratios, X-ray to IR spectral slopes, and optical morphologies. Although we have spectroscopic redshifts for only a minority of the Chandra sources (10 type 1 QSOs or Seyfert sources and three galaxies), the available SEDs constrain the redshifts for most of the sample sources, which turn out to be typically at 0.5