Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Universal Value-Function Uncertainties

Moritz A. Zanger, Max Weltevrede, Yaniv Oren, Pascal R. Van der Vaart, Caroline Horsch, Wendelin Böhmer, and Matthijs T. J. Spaan. Universal Value-Function Uncertainties. In Proc. Int. Conf. on Learning Representations, 2026.

Download

pdf 

Abstract

Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional value uncertainty, incorporating the future uncertainties any policies may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.

BibTeX Entry

@InProceedings{Zanger26uvuICLR,
  author =       {Zanger, Moritz A. and Weltevrede, Max and Oren,
                  Yaniv and Van der Vaart, Pascal R. and Horsch,
                  Caroline and B{\"o}hmer, Wendelin and Spaan,
                  Matthijs T. J.},
  title =        {Universal Value-Function Uncertainties},
  booktitle =    {Proc. Int. Conf. on Learning Representations},
  year =         2026
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC