Polarimetry constitutes one of the fundamental tools for characterizing the surface texture and composition of airless Solar System bodies. In 2006, polarimetric observations led to the discovery of a new type of asteroids, which displays a peculiar polarimetric response. These asteroids are collectively known as "Barbarians", from (234) Barbara the first discovered one.The most commonly accepted explanation for this perculiar polarization response seems to be the presence of a high percentage of fluffy-type Calcium Aluminium-rich Inclusions (CAIs), whose optical properties could produce the observed polarization. Their reflectance spectra also exibit an absorption feature in the near-infrared around 2.1-2.2 microns, that is characteristic of this peculiar group.Based on these results, we organized a systematic polarimetric and near-infrared observational campaign of known Barbarians or candidate asteroids. These campaigns include members of the family of 1040 Klumpkea, 2085 Henan and 729 Watsonia, which are known to contain Barbarian and/or L-type asteroids also suspected to have such a polarimetric behaviour. We have made use of the ToPo polarimeter at the 1m telescope of the Centre pédagogique Planète et Univers (C2PU, Observatoire de la Côte d'Azur, France). The spectroscopic observations in the near-infrared were obtained with the SpeX instrument at the NASA's InfraRed Telescope Facility (IRTF).By combining polarimetry and spectroscopy we find a correlation between the abundance of CAIs and the inversion angle of the phase-polarization curve of Barbarian asteroids. This is the first time that a direct link has been established between a specific polarimetric response and the surface composition of asteroids. In addition, we find a considerable variety of CAI abundance from one object to the other, consistent with a wide range of possible albedos. Since these asteroids constitute a reservoir of primitive Solar System material, understanding their origin can shed light on the processes driving the formation and transport of the refractory minerals that first condensed in the protoplanetary disk.