Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

An NTK Theory Approach to UCB Estimation in Semi-gradient TD Learning

Yijun Wu, Pascal R. Van der Vaart, Moritz A. Zanger, and Matthijs T. J. Spaan. An NTK Theory Approach to UCB Estimation in Semi-gradient TD Learning. In Proc. of Uncertainty in Artificial Intelligence, pp. 7403–7415, 2026.

Download

pdf 

Abstract

Efficient exploration is a central challenge in reinforcement learning (RL), especially under sparse rewards, large state-action spaces, or adversarial environment dynamics. A principled approach to exploration is to choose actions that maximize the upper bound return. Under certain conditions, this can be formulated in the Bayesian framework as exploration based on the posterior variance. In practice, the predictive variance of a neural network is often used as a surrogate for the posterior variance by techniques such as ensembles, randomized priors, and random network distillation. However, for wide neural networks, neural tangent kernel (NTK) theory proves there is a discrepancy between the predictive variance of a wide neural network trained by gradient descent and its posterior variance. This is a known issue in supervised learning and there exist methods to reconcile this discrepancy. In this work, we show that, in contrast to the supervised setting, reconciliation is not possible for semi-gradient temporal difference (TD) learning, which is a commonly used algorithm in deep RL. Instead, we derive the upper confidence bound (UCB) of the value function in semi-gradient TD learning directly and show that this UCB can be estimated by a neural network. To aid our analysis, we extend NTK theory to regularized semi-gradient TD learning.

BibTeX Entry

@InProceedings{Wu26uai,
  author =       {Wu, Yijun and Van der Vaart, Pascal R. and Zanger,
                  Moritz A. and Spaan, Matthijs T. J.},
  title =        {An {NTK} Theory Approach to {UCB} Estimation in
                  Semi-gradient {TD} Learning},
  booktitle =    {Proc. of Uncertainty in Artificial Intelligence},
  pages =        {7403--7415},
  year =         2026
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC