Context: Inversion techniques are the most powerful methods to obtain information about the thermodynamical and magnetic properties of solar and stellar atmospheres. In the recent years, we have witnessed the development of highly sophisticated inversion codes that are now widely applied to spectro-polarimetric observations. The majority of these inversion codes are based on the optimization of a complicated non-linear merit function. The experience gained has facilitated the recovery of the model that best fits a given observation. However, and except for the recently developed inversion codes based on database search algorithms together with the application of Principal Component Analysis, no reliable and statistically well-defined confidence intervals can be obtained for the parameters inferred from the inversions. Aims: A correct estimation of the confidence intervals for all the parameters that describe the model is mandatory. Additionally, it is fundamental to apply efficient techniques to assess the ability of models to reproduce the observations and to determine to what extent the models have to be refined or can be simplified. Methods: Bayesian techniques are applied to analyze the performance of the model to fit a given observed Stokes vector. The posterior distribution, that takes into account both the information about the priors and the likelihood, is efficiently sampled using a Markov chain Monte Carlo method. For simplicity, we focus on the Milne-Eddington approximate solution of the radiative transfer equation and we only take into account the generation of polarization through the Zeeman effect. However, the method is extremely general and other more complex forward models can be applied, even allowing for the presence of atomic polarization. Results: We illustrate the method with different problems, from academic to more realistic examples. We show that the information provided by the posterior distribution is fundamental to understand and determine the amount of information available in the Stokes profiles in these particular cases. Appendix A and B are only available in electronic form at http://www.aanda.org.