Context: Eight accreting millisecond X-ray pulsars (AMXPs) are known to date. Although these systems are well studied at high energies, very little information is available for their optical/NIR counterparts. Up to now, only two of them, SAX J1808.4-3658 and IGR J00291+5934, have a secure multi-band detection of their optical counterparts in quiescence. Aims: All these systems are transient low-mass X-ray binaries. Optical and NIR observations carried out during quiescence give a unique opportunity to constrain the nature of the donor star and to investigate the origin of the observed quiescent luminosity at long wavelengths. In addition, optical observations can be fundamental as they ultimately allow us to estimate the compact object mass through mass function measurements. Methods: Using data obtained with the ESO-Very Large Telescope, we performed a deep optical and NIR photometric study of the fields of XTE J1814-338 and of the ultracompact systems XTE J0929-314 and XTE J1807-294 during quiescence in order to look for the presence of a variable counterpart. If suitable candidates were found, we also carried out optical spectroscopy. Results: We present here the first multi-band (VR) detection of the optical counterpart of XTE J1814-338 in quiescence together with its optical spectrum. The optical light curve shows variability in both bands consistent with a sinusoidal modulation at the known 4.3 h orbital period and presents a puzzling decrease of the V-band flux around superior conjunction that may be interpreted as a partial eclipse. The marginal detection of the very faint counterpart of XTE J0929-314 and deep upper limits for the optical/NIR counterpart of XTE J1807-294 are also reported. We also briefly discuss the results reported in the literature for the optical/NIR counterpart of XTE J1751-305. Conclusions: Our findings are consistent with AMXPs being systems containing an old, weakly magnetized neutron star, reactivated as a millisecond radio pulsar during quiescence which irradiates the low-mass companion star. The absence of type I X-ray bursts and of hydrogen and helium lines in outburst spectra of ultracompact (P_orb <1 h) AMXPs suggests that the companion stars are likely evolved dwarf stars. Based on observations made with ESO Telescopes at the Paranal Observatory under programmes ID 077.D-0677(D) and ID 079.D-0884(A) and observations made with the ESO telescopes obtained from the ESO/ST-ECF Science Archive Facility.