Publications

Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic Exploration

Guopeng Li, Matthijs T. J. Spaan, and Julian F. P. Kooij. Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic Exploration. In Proc. Int. Conf. on Learning Representations, 2026.

Download

pdf

Abstract

When formulating safety as limits of cumulative cost, safe reinforcement learning (RL) learns policies that maximize rewards subject to these constraints during both data collection and deployment. While off-policy methods offer high sample efficiency, their application to safe RL faces substantial challenges from constraint violations caused by the cost-agnostic exploration and the underestimation bias in the cost value function. To address these challenges, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy primal-dual safe RL method that integrates cost-bounded exploration and conservative distributional RL. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space, and adaptively adjusts the trust region to control constraint violation in exploration. Second, we adopt truncated quantile critics to mitigate the underestimation bias in costs. The quantile critics also quantify distributional, risk-sensitive epistemic uncertainty for guiding exploration. Experiments across velocity-constrained robot locomotion, safe navigation, and complex autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive safety performance during evaluation, and controlled data collection cost in exploration. The results highlight the proposed method as a promising solution for safety-critical RL.

BibTeX Entry

@InProceedings{Li26iclr,
  author =       {Guopeng Li and Matthijs T. J. Spaan and Julian
                  F. P. Kooij},
  title =        {Off-Policy Safe Reinforcement Learning with
                  Cost-Constrained Optimistic Exploration},
  booktitle =    {Proc. Int. Conf. on Learning Representations},
  year =         2026
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC