Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Epistemic Bellman Operators

Pascal R. Van der Vaart, Matthijs T. J. Spaan, and Neil Yorke-Smith. Epistemic Bellman Operators. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 20973–20981, 2025.

Download

pdf HTML 

Abstract

Uncertainty quantification remains a difficult challenge in reinforcement learning. Several algorithms exist that successfully quantify uncertainty in a practical setting. However it is unclear whether these algorithms are theoretically sound and can be expected to converge. Furthermore, they seem to treat the uncertainty in the target parameters in different ways. In this work, we unify several practical algorithms into one theoretical framework by defining a new Bellman operator on distributions, and show that this Bellman operator is a contraction. We highlight use cases of our framework by analyzing an existing Bayesian Q-learning algorithm, and also introduce a novel uncertainty-aware variant of PPO that adaptively sets its clipping hyperparameter.

BibTeX Entry

@InProceedings{VanDerVaart25aaai,
  author =       {Van der Vaart, Pascal R. and Spaan, Matthijs
                  T. J. and Yorke-Smith, Neil},
  title =        {Epistemic {B}ellman Operators},
  year =         2025,
  booktitle =    {Proceedings of the AAAI Conference on Artificial
                  Intelligence},
  pages =        {20973--20981}
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC