Asteroid 162173 Ryugu is a carbonaceous asteroid that was visited by Japan's Hayabusa2 spacecraft in 2018. The formation mechanism of the "spinning-top" shape of Ryugu is a vital clue to the dynamical history of the near-Earth asteroid. In this study, we address the long-term evolution of its spin state induced by the Yarkovsky-O'Keefe-Radzievskii-Paddack (YORP) effect, that is, the thermal recoil torque that changes the rotation period and spin-pole direction. Given the current orbit, spin state, and three-dimensional shape of Ryugu observed by Hayabusa2, we computed the YORP torque exerted on Ryugu using a simplified thermal model assuming zero thermal conductivity. Despite variations in the meter-scaled topography, all 20 shape models that we examined indicate that the spin velocity of Ryugu is currently decreasing at a rate of >(-6.3 to -0.42>)×10-6 deg/day2. Our findings also suggest that the thermal torque is responsible for maintaining the spin pole upright with respect to the orbital plane. Therefore, the YORP effect may explain the significant spin-down from an earlier period of 3.5 hr to the present period of 7.6 hr. The corresponding time scale of the spin-down is estimated to be 0.58-8.7 million years, depending on the input shape models. This time scale is comparable to the formation period of the largest crater, Urashima (5-12 Ma), or the western bulge (2-9 Ma) as derived from previous studies on crater statistics in Ryugu. Thus, its rotation may have started to decelerate as a consequence of major resurfacing events.