Context. The physical conditions of the solar photosphere change on very small spatial scales both horizontally and vertically. Such a complexity may pose a serious obstacle to the accurate determination of solar magnetic fields. Aims: We examine the applicability of Milne-Eddington (ME) inversions to high spatial resolution observations of the quiet Sun. Our aim is to understand the connection between the ME inferences and the actual stratifications of the atmospheric parameters. Methods: We use magnetoconvection simulations of the solar surface to synthesize asymmetric Stokes profiles such as those observed in the quiet Sun. We then invert the profiles with the ME approximation. We perform an empirical analysis of the heights of formation of ME measurements and analyze the uncertainties brought about by the ME approximation. We also investigate the quality of the fits and their relationship with the model stratifications. Results: The atmospheric parameters derived from ME inversions of high-spatial resolution profiles are reasonably accurate and can be used for statistical analyses of solar magnetic fields, even if the fit is not always good. We also show that the ME inferences cannot be assigned to a specific atmospheric layer: different parameters sample different ranges of optical depths, and even the same parameter may trace different layers depending on the physical conditions of the atmosphere. Despite this variability, ME inversions tend to probe deeper layers in granules than in intergranular lanes. Figure 10 and appendix are only available in electronic form at http://www.aanda.org