Publications

On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

Moritz A. Zanger, Yijun Wu, Pascal R. Van der Vaart, Wendelin Böhmer, and Matthijs T. J. Spaan. On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference. In Proc. of Uncertainty in Artificial Intelligence, pp. 7952–7976, 2026.

Download

pdf

Abstract

Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g., Bayesian inference or deep ensembles. We establish these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND---its squared self-predictive error---is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of Bayesian inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact Bayesian posterior predictive distribution using this modified Bayesian RND model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and Bayesian inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.

BibTeX Entry

@InProceedings{Zanger26uai,
  author =       {Zanger, Moritz A. and Wu, Yijun and Van der Vaart,
                  Pascal R. and B{\"o}hmer, Wendelin and Spaan,
                  Matthijs T. J.},
  title =        {On the Equivalence of Random Network Distillation,
                  Deep Ensembles, and {B}ayesian Inference},
  booktitle =    {Proc. of Uncertainty in Artificial Intelligence},
  pages =        {7952--7976},
  year =         2026
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC