Context. The quiet Sun magnetic fields produce ubiquitous bright points (BPs) that cover a significant fraction of the solar surface. Their contribution to the total solar irradiance (TSI) is so-far unknown. Aims: We aim at measuring the center-to-limb variation (CLV) of the fraction of solar surface covered by quiet Sun magnetic bright points. The fraction is referred to as the fraction of covered surface (FCS). Methods: We count the area covered by BPs in G-band images obtained at various heliocentric angles with the 1-m Swedish Solar Telescope on La Palma. We restore the images to bring them close to the diffraction limit of the instrument (~0'.1). Results: The FCS is largest at the disk center (≃1%), and then drops down to become ≃0.2% at μ ≃ 0.3 (where μ is the cosine of the heliocentric angle). The relationship has a large scatter, which we evaluate by comparing different subfields within our FOVs. We develop a toy-model to describe the observed CLV, which considers the BPs as depressions in the mean solar photosphere characterized by a depth, a width, and a spread in the inclinations. Although the model is poorly constrained by observations, it shows the BPs to be shallow structures (depth < width) with a large range of inclinations. We also estimate how different parts of the solar disk may contribute to the TSI variations, finding that 90% is contributed by BPs with μ > 0.5, and half of it is due to BPs with μ > 0.8.