Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection

Guopeng Li, Moritz A. Zanger, Matthijs T. J. Spaan, and Julian F. P. Kooij. COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection. arXiv:2606.04749, 2026.

Download

pdf 

Abstract

Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.

BibTeX Entry

@Misc{Li26copq,
  author =       {Guopeng Li and Moritz A. Zanger and Matthijs
                  T. J. Spaan and Julian F. P. Kooij},
  title =        {{COP-Q}: Safety-First Reinforcement Learning for
                  Robot Control via {C}holesky-Ordered Projection},
  howpublished = {arXiv:2606.04749},
  year =         2026
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC