We derive the stellar population parameters of 11 quiescent ultra-diffuse galaxies (UDGs) from Keck/KCWI data. We supplement these with 14 literature UDGs, creating the largest spectroscopic sample of UDGs to date (25). We find a strong relationship between their α-enhancement and their star formation histories: UDGs that formed on very short time-scales have elevated [Mg/Fe] abundance ratios, whereas those forming over extended periods present lower values. Those forming earlier and faster are overall found in high-density environments, being mostly early infalls into the cluster. No other strong trends are found with infall times. We analyse the stellar mass-metallicity, age-metallicity, and [Mg/Fe]-metallicity relations of the UDGs, comparing them to other types of low mass galaxies. Overall, UDGs scatter around the established stellar mass-metallicity relations of classical dwarfs. We find that GC-rich UDGs have intermediate-to-old ages, but previously reported trends of galaxy metallicity and GC richness are not reproduced with this spectroscopic sample due to the existence of GC-rich UDGs with elevated metallicities. In addition, we also find that a small fraction of UDGs could be 'failed-galaxies', supported by their GC richness, high alpha-abundance, fast formation time-scales and that they follow the mass-metallicity relation of z ~2 galaxies. Finally, we also compare our observations to simulated UDGs. We caution that there is not a single simulation that can produce the diverse UDG properties simultaneously, in particular the low metallicity failed galaxy like UDGs.