We present multiband optical/IR photometry of V4580 Sgr, the optical counterpart of the accretion-powered millisecond pulsar SAX J1808.4-3658, taken during the 1998 X-ray outburst of the system. The optical flux is consistent with emission from an X-ray-heated accretion disk. Self-consistent modeling of the X-ray and optical emission during the outburst yields a best-fit extinction of AV=0.68+0.37-0.15 and an inclination of cosi=0.65+0.23-0.33 (90% confidence), assuming a distance of 2.5 kpc. This inclination range requires that the stellar companion of the pulsar has extremely low mass, Mc=0.05-0.10 Msolar. Some of the IR observations are inconsistent with disk emission and are too bright to be from either the disk or the companion, even in the presence of X-ray heating.