Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Scaling Safe Policy Improvement: Monte Carlo Tree Search and Policy Iteration Strategies

Federico Bianchi, Alberto Castellini, Edoardo Zorzi, Thiago D. Simão, Matthijs T. J. Spaan, and Alessandro Farinelli. Scaling Safe Policy Improvement: Monte Carlo Tree Search and Policy Iteration Strategies. Journal of Artificial Intelligence Research, 84, 2025.

Download

pdf 

Abstract

Offline Reinforcement Learning (RL) allows policies to be trained on pre-collected datasets without requiring additional interactions with the environment. This approach bypasses the need for real-time data acquisition in real-world applications, which can be impractical due to the safety issues inherent in the learning process. However, offline RL faces significant challenges, such as distributional shifts and extrapolation errors, and the resulting policies might underperform compared to the baseline policy. Safe policy improvement algorithms mitigate these issues, enabling the reliable deployment of RL approaches in real-world scenarios where historical data is available, guaranteeing that any policy changes will not result in worse performance compared to the baseline policy used to collect training data. In this paper, we propose MCTS-SPIBB, an algorithm that leverages Monte Carlo Tree Search (MCTS) for scaling safe policy improvement to large domains. We theoretically prove that the policy generated by MCTS-SPIBB converges to the optimal safely improved policy produced by Safe Policy Improvement with Baseline Bootstrapping (SPIBB) as the number of simulations increases. Additionally, we introduce SDP-SPIBB, a novel extension of SPIBB designed to address the scalability limitations of the standard algorithm via Scalable Dynamic Programming. Our empirical analysis across four benchmark domains demonstrates that MCTS-SPIBB and SDP-SPIBB significantly enhance the scalability of safe policy improvement, providing robust and efficient algorithms for large-scale applications. These contributions represent a significant step towards the deployment of safe RL algorithms in complex real-world environments.

BibTeX Entry

@Article{Bianchi25,
  author =       {Federico Bianchi and Alberto Castellini and Edoardo
                  Zorzi and Thiago D. Sim{\~a}o and Matthijs
                  T. J. Spaan and Alessandro Farinelli},
  title =        {Scaling Safe Policy Improvement: {M}onte {C}arlo
                  Tree Search and Policy Iteration Strategies},
  journal =      {Journal of Artificial Intelligence Research},
  volume =       84,
  year =         2025
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC