We present the optimization procedure of the baffle mounted on the GroundBIRD telescope for measuring the polarization of the Cosmic microwave background (CMB). The telescope employs a dual mirror reflective telescope installed in a cryostat. The primary objectives were to minimize stray light contamination, maintain the integrity of the main beam, and ensure that thermal loading from the baffle remains significantly below that from the atmosphere. Using quasi-optical simulations, we have optimized the baffles aperture angle to suppress stray light without degrading the main beam quality. We confirmed through Moon observations that the optimized baffle design works to eliminate the contamination of the stray light as expected. Furthermore, no measurable degradation in the noise equivalent temperature was detected, indicating minimal thermal impact. These results show that our baffle optimization strategy effectively reduces systematic errors while maintaining observational sensitivity, providing valuable insights for future CMB experiments with similar optical architectures.