Aims: In a companion paper we have presented many products derived from the application of the spectral synthesis code starlight to datacubes from the CALIFA survey, including 2D maps of stellar population properties (such as mean ages, mass, and extinction) and 1D averages in the temporal and spatial dimensions. Our goal here is to assess the uncertainties in these products. Methods: Uncertainties associated to noise and spectral shape calibration errors in the data and to the synthesis method were investigated by means of a suite of simulations, perturbing spectra and processing them through our analysis pipelines. The simulations used 1638 CALIFA spectra for NGC 2916, with perturbation amplitudes gauged in terms of the expected errors. A separate study was conducted to assess uncertainties related to the choice of evolutionary synthesis models, the key ingredient in the translation of spectroscopic information into stellar population properties. We compare the results obtained with three different sets of models: the traditional Bruzual & Charlot models, a preliminary update of them, and a combination of spectra derived from the Granada and MILES models. About 105 spectra from over 100 CALIFA galaxies were used in this comparison. Results: Noise and shape-related errors at the level expected for CALIFA propagate to uncertainties of 0.10-0.15 dex in stellar masses, mean ages, and metallicities. Uncertainties in AV increase from 0.06 mag for random noise to 0.16 mag for spectral shape errors. Higher-order products such as star formation histories are more uncertain than global properties, but still relatively stable. Owing to the large number statistics of datacubes, spatial averaging reduces uncertainties while preserving information on the history and structure of stellar populations. Radial profiles of global properties, and star formation histories averaged over different regions are much more stable than those obtained for individual spaxels. Uncertainties related to the choice of base models are larger than those associated with data and method. Except for metallicities, which come out very different when fits are performed with the Bruzual & Charlot models, differences in mean age, mass, and metallicity are of the order of 0.15 to 0.25 dex, and 0.1 mag for AV. Spectral residuals are of the order of 1% on average, but with systematic features of up to 4% amplitude. We discuss the origin of these features, most of which are present in both in CALIFA and SDSS spectra.