Millisecond radio pulsars are neutron stars that have been spun up by the transfer of angular momentum during the low-mass X-ray binary phase. The transition from an accretion-powered pulsar to a rotation-powered pulsar takes place on evolutionary timescales at the end of the accretion process; however, it may also occur sporadically in systems undergoing transient X-ray activity. We have obtained the first optical spectrum of the low-mass transient X-ray pulsar SAX J1808.4-3658 in quiescence. Similar to the black widow millisecond pulsar B1957+20, this X-ray pulsar shows a large optical modulation at the orbital period due to an irradiated companion star. Using the brightness of the companion star as a bolometer, we conclude that a very high irradiating luminosity, a factor of ~100 larger than directly observed, must be present in the system. This most likely derives from a rotation-powered neutron star that resumes activity during quiescence.