We present the first determination of the proper motion of the neutron star low mass X-ray binary Cen X-4 measured from relative astrometry of the secondary star using optical images at different epochs. We determine the Galactic space velocity components of the system and find them to be significantly different from the mean values that characterize the kinematics of stars belonging to the halo, and the thin and the thick disc of the Galaxy. The high metallicity of the secondary star of the system rules out a halo origin and indicates that the system probably originated in the Galactic disc. A statistical analysis of the galactocentric motion revealed that this binary moves in a highly eccentric (e≃ 0.85±0.1) orbit with an inclination of ≃110° to the Galactic plane. The large Galactic space velocity components strongly support that a high natal kick as a result of a supernova explosion could have propelled the system into such an orbit from a birth place in the Galactic disc. The high Li abundance in the secondary, comparable to that of stars in star forming regions and young stellar clusters like the Pleiades, may suggest a relatively recent formation of the system. Following the orbit backwards in time, we found that the system could have been in the inner regions of the Galactic disc ~100-200 Myr ago. The neutron star might have formed at that moment. However, we cannot rule out the possibility that the system formed at a much earlier time if a Li production mechanism exists in this LMXB.