Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Value Improved Actor Critic Algorithms

Yaniv Oren, Moritz A. Zanger, Pascal R. Van der Vaart, Mustafa Mert Çelikok, Wendelin Böhmer, and Matthijs T. J. Spaan. Value Improved Actor Critic Algorithms. In Advances in Neural Information Processing Systems, pp. 51608–51644, 38, 2025.

Download

pdf 

Abstract

To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow and steady changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To address this tradeoff, we propose to extend the standard framework of actor critic algorithms with value-improvement: a second greedification operator applied only when updating the policy's value estimate. In this framework the agent can evaluate non-parameterized policies and perform much greedier updates while maintaining the steady gradient-based improvement to the parameterized acting policy. We prove that this approach converges in the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain. Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.

BibTeX Entry

@InProceedings{Oren25viac,
  author =       {Oren, Yaniv and Zanger, Moritz A. and Van der
                  Vaart, Pascal R. and {\c C}elikok, Mustafa Mert and
                  B{\"o}hmer, Wendelin and Spaan, Matthijs T. J.},
  title =        {Value Improved Actor Critic Algorithms},
  booktitle =    {Advances in Neural Information Processing Systems},
  pages =        {51608--51644},
  volume =       38,
  year =         2025
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Fri Aug 28, 2026 12:56:06 UTC