The evolution and fate of very massive stars (VMS) is tightly connected to their mass-loss properties. Their initial and final masses differ significantly as a result of mass loss. VMS have strong stellar winds and extremely high ionising fluxes, which are thought to be critical sources of both mechanical and radiative feedback in giant H ii regions. However, how VMS mass-loss properties change during stellar evolution is poorly understood. In the framework of the VLT-Flames Tarantula Survey (VFTS), we explore the mass-loss transition region from optically thin O star winds to denser WNh Wolf-Rayet star winds, thereby testing theoretical predictions. To this purpose we select 62 O, Of, Of/WN, and WNh stars, an unprecedented sample of stars with the highest masses and luminosities known. We perform a spectral analysis of optical VFTS as well as near-infrared VLT/SINFONI data using the non-LTE radiative transfer code CMFGEN to obtain both stellar and wind parameters. For the first time, we observationally resolve the transition between optically thin O star winds and optically thick hydrogen-rich WNh Wolf-Rayet winds. Our results suggest the existence of a "kink" between both mass-loss regimes, in agreement with recent Monte Carlo simulations. For the optically thick regime, we confirm the steep dependence on the classical Eddington factor Γe from previous theoretical and observational studies. The transition occurs on the main sequence near a luminosity of 106.1L⊙, or a mass of 80 ... 90 M⊙. Above this limit, we find that - even when accounting for moderate wind clumping (with fv = 0.1) - wind mass-loss rates are enhanced with respect to standard prescriptions currently adopted in stellar evolution calculations. We also show that this results in substantial helium surface enrichment. Finally, based on our spectroscopic analyses, we are able to provide the most accurate ionising fluxes for VMS known to date, confirming the pivotal role of VMS in ionising and shaping their environments. Appendices are available in electronic form at http://www.aanda.org