Context. Full Stokes filter-polarimeters are key instruments for investigating the rapid evolution of magnetic structures on the solar surface. To this end, the image quality is routinely improved using a-posteriori image reconstruction methods. Aims: We analyze the robustness of circular polarization asymmetries to phase-diversity image reconstruction techniques. Methods: We used snapshots of magneto-hydrodynamical simulations carried out with different initial conditions to synthesize spectra of the magnetically sensitive Fe i line at 5250.2 Å. We degraded the synthetic profiles spatially and spectrally to simulate observations with the IMaX full Stokes filter-polarimeter. We also simulated the focused/defocused pairs of images used by the phase-diversity algorithm for reconstruction and the polarimetric modulation scheme. We assume that standard optimization methods are able to infer the projection of the wavefront on the Zernike polynomials with 10% precision. We also consider the less favorable case of 25% precision. We obtain reconstructed monochromatic modulated images that are later demodulated and compared with the original maps. Results: Although asymmetries are often difficult to define in the quiet Sun due to the complexity of the Stokes V profiles, we show how asymmetries are degraded with spatial and spectral smearing. The results indicate that, although image reconstruction techniques reduce the spatial smearing, they can modify the asymmetries of the profiles, which are mainly caused by the appearance of spatially-correlated noise.