We present deep 3.6-8 μm imaging of the Hubble Deep Field-South with the Infrared Array Camera (IRAC) on the Spitzer Space Telescope. We study distant red galaxies (DRGs) at z>2 selected by Js-Ks>2.3 and compare them with a sample of Lyman break galaxies (LBGs) at z=2-3. The observed UV-to-8 μm spectral energy distributions are fitted with stellar population models to constrain star formation histories and derive stellar masses. We find that 70% of the DRGs are best described by dust-reddened star-forming models and 30% are very well fitted with old and ``dead'' models. Using only the I-Ks and Ks-4.5 μm colors, we can effectively separate the two groups. The dead systems are among the most massive at z~2.5 (mean stellar mass = 0.8×1011 Msolar) and likely formed most of their stellar mass at z>5. To a limit of 0.5×1011 Msolar, their number density is ~10 times lower than that of local early-type galaxies. Furthermore, we use the IRAC photometry to derive rest-frame near-infrared J, H, and K fluxes. The DRGs and LBGs together show a large variation (a factor of 6) in the rest-frame K-band mass-to-light ratios (M/LK), implying that even a Spitzer 8 μm-selected sample would be very different from a mass-selected sample. The average M/LK of the DRGs is about 3 times higher than that of the LBGs, and DRGs dominate the high-mass end. The M/LK values and ages of the two samples appear to correlate with derived stellar mass, with the most massive galaxies being the oldest and having the highest mass-to-light ratios, similar to what is found in the low-redshift universe. Based on observations made with the Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory, California Institute of Technology, under NASA contract 1407. Support for this work was provided by NASA through contract 125790 issued by JPL/Caltech. Based on service-mode observations collected at the European Southern Observatory, Paranal, Chile (program 164.O-0612). Based on observations with the NASA/ESA Hubble Space Telescope, obtained at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS 5-26555