Context. Over the past 20 yr, the quietest areas of the solar surface have revealed a weak but extremely dynamic magnetism occurring at small scales (<500 km), which may provide an important contribution to the dynamics and energetics of the outer layers of the atmosphere. Understanding this magnetism requires the inference of physical quantities from high-sensitivity spectro-polarimetric data with high spatio-temporal resolution. Aims: We present high-precision spectro-polarimetric data with high spatial resolution (0.4'') of the very quiet Sun at 1.56 μm obtained with the GREGOR telescope to shed some light on this complex magnetism. Methods: We used inversion techniques in two main approaches. First, we assumed that the observed profiles can be reproduced with a constant magnetic field atmosphere embedded in a field-free medium. Second, we assumed that the resolution element has a substructure with either two constant magnetic atmospheres or a single magnetic atmosphere with gradients of the physical quantities along the optical depth, both coexisting with a global stray-light component. Results: Half of our observed quiet-Sun region is better explained by magnetic substructure within the resolution element. However, we cannot distinguish whether this substructure comes from gradients of the physical parameters along the line of sight or from horizontal gradients (across the surface). In these pixels, a model with two magnetic components is preferred, and we find two distinct magnetic field populations. The population with the larger filling factor has very weak ( 150 G) horizontal fields similar to those obtained in previous works. We demonstrate that the field vector of this population is not constrained by the observations, given the spatial resolution and polarimetric accuracy of our data. The topology of the other component with the smaller filling factor is constrained by the observations for field strengths above 250 G: we infer hG fields with inclinations and azimuth values compatible with an isotropic distribution. The filling factors are typically below 30%. We also find that the flux of the two polarities is not balanced. From the other half of the observed quiet-Sun area 50% are two-lobed Stokes V profiles, meaning that 23% of the field of view can be adequately explained with a single constant magnetic field embedded in a non-magnetic atmosphere. The magnetic field vector and filling factor are reliable inferred in only 50% based on the regular profiles. Therefore, 12% of the field of view harbour hG fields with filling factors typically below 30%. At our present spatial resolution, 70% of the pixels apparently are non-magnetised.