Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning

Pascal R. Van der Vaart, Neil Yorke-Smith, and Matthijs T. J. Spaan. Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning. arXiv:2508.21488, 2025.

Download

pdf 

Abstract

Uncertainty quantification in reinforcement learning can greatly improve exploration and robustness. Approximate Bayesian approaches have recently been popularized to quantify uncertainty in model-free algorithms. However, so far the focus has been on improving the accuracy of the posterior approximation, instead of studying the accuracy of the prior and likelihood assumptions underlying the posterior. In this work, we demonstrate that there is a cold posterior effect in Bayesian deep Q-learning, where contrary to theory, performance increases when reducing the temperature of the posterior. To identify and overcome likely causes, we challenge common assumptions made on the likelihood and priors in Bayesian model-free algorithms. We empirically study prior distributions and show through statistical tests that the common Gaussian likelihood assumption is frequently violated. We argue that developing more suitable likelihoods and priors should be a key focus in future Bayesian reinforcement learning research and we offer simple, implementable solutions for better priors in deep Q-learning that lead to more performant Bayesian algorithms.

BibTeX Entry

@Misc{VanDerVaart25priors,
  author =       {Van der Vaart, Pascal R. and Yorke-Smith, Neil and
                  Spaan, Matthijs T. J.},
  title =        {Priors Matter: Addressing Misspecification in {B}ayesian Deep {Q}-Learning},
  howpublished = {arXiv:2508.21488},
  year =         2025
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC