Context. The radial velocity (RV) method plays a key role in modern-day astrophysics. One of the most common techniques for extracting precise RVs from state-of-the-art spectrographs is template-matching (TM) algorithms. They have been shown to perform better than a cross-correlation function (CCF) approach in cases of cooler stars (e.g. M dwarfs) and multiple implementations have appeared over the past years. More recently, line-by-line (LBL) approaches offer an alternative avenue to extract RVs by analyzing individual spectral lines. Aims. In this paper, we identify and explore a previously unidentified, multi-meter-per-second, systematic correlation between time and RVs inferred through TM and LBL methods. We evaluate the influence of the data-driven stellar template in the RV bias and hypothesise on the possible sources of this effect. Methods. We used the s-BART pipeline to extract RVs from three different datasets gathered over four nights of ESPRESSO and HARPS observations. We demonstrate that the effect can be recovered on a larger sample of 19 targets, totalling 4124 ESPRESSO observations over 38 nights. We also showcase the presence of the bias in RVs extracted with the SERVAL and ARVE pipelines. Lastly, we explore the construction of the stellar template over the five years of ESPRESSO observations of HD 10700, totalling more than 2000 observations. Results. We find that a systematic quasi-linear bias affects the RV extraction with slopes that vary from —0.3 ms‑1 h‑1to —52 m s‑1 h‑1 in our sample. This trend is not observed in CCF RVs and only appears when all observations of a given star are collected within a short time period (timescales of hours). We show that this systematic contamination exists in the RV time series of two different template-matching pipelines and one line-by-line pipeline, and it is agnostic to the spectrograph. We also find that this effect is linked to the construction of the stellar template, as we were able to mitigate it through a careful selection of the observations used to construct it. Our results suggest that a contamination of micro-telluric features, coupled with other sources of correlated noise, could be the driving factor of this effect. We also show that this effect does not impact the usual usage of template-matching for the detection and characterisation of exoplanets. However, the short-timescale science cases, such as asteroseismology as well as transit and atmospheric characterisation, can be severely affected.