Improving Value Estimation Critically Enhances Vanilla Policy Gradient

Tao Wang, Ruipeng Zhang, Sicun Gao · 2025

Read Improving Value Estimation Critically Enhances Vanilla Policy Gradient by Tao Wang, Ruipeng Zhang, Sicun Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19247.txt.

Introduction

Tao Wang, Ruipeng Zhang, and Sicun Gao question a common belief about how to train AI by trial and reward, in a 2025 paper. Prize methods like PPO beat the plain policy gradient method on many tasks, and credit is usually given to their trust regions, guardrails that keep each update small. The team suspect the real edge lies elsewhere.

PPO-style methods also update their value estimates, their guesses of future reward, many more times per round. When the authors give the plain method the same extra value updates, it matches or beats PPO on all standard control test tasks. The boosted plain method also shrugs off knob settings that break the fancier methods. The lesson is plain: better value guesses, not guardrails, may drive most of the gain.

Catalog record

CREATORTao Wang, Ruipeng Zhang, Sicun Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Value Estimation Critically Enhances Vanilla Policy Gradient Tao Wang, Ruipeng Zhang, Sicun Gao Abstract Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use. 1 Introduction Deep policy gradient methods have demonstrated their strength in various domains, including robot learning ( Song et al., 2023 ) , game playing ( Vinyals et al., 2019 ) , and large language model training ( Ouyang et al., 2022 ) . Policy gradient methods in its modern versions, such as Trust Region Policy Optimization (TRPO) ( Schulman et al., 2015a ) , Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , have been the popular choice for delivering state-of-the-art performance across the broad range of RL tasks. In comparison, the original policy gradient algorithm, Vanilla Policy Gradient (VPG) ( Sutton et al., 1999 ) , typically performs significantly worse than the modern algorithms. The common explanation for why TRPO and PPO outperform VPG is their ability to prevent excessive large policy updates. This is achieved through a surrogate objective that incorporates importance sampling, constrained by a limit on policy step size each iteration. While monotonic improvement is theoretically guaranteed for sufficiently small policy updates ( Kakade & Langford, 2002 ; Schulman et al., 2015a ) , empirical results suggest otherwise: smaller learning rates for the policy network do not always yield better performance ( Andrychowicz et al., 2021 ) . According to the trust region theory ( Schulman et al., 2015a ) , there is a gap between theory and practice because the theory guarantees improvement of the original policy objective only when the surrogate objective improves by a margin greater than the distance between the old and new policies. However, there is no guarantee that this condition is consistently satisfied in practical policy training. Moreover, it has been reported that deep policy gradient methods often suffer from implementation issues such as brittleness, poor reproducibility, and sensitivity to hyperparameter choices ( Henderson et al., 2018 ; Engstrom et al., 2020 ; Andrychowicz et al., 2021 ) . Extensive empirical analysis has shown that the behavior of deep policy gradient methods does not align with the predictions of their motivating framework ( Ilyas et al., 2020 ) . Recent studies further highlight that the optimization landscape of many RL tasks is highly non-smooth and even fractal, raising questions about the well-posedness of all policy gradient methods ( Wang et al., 2023 ) . Consequently, there is still no clear answer to the question: while almost all theoretical assumptions are violated in practice, why does PPO perform better than vanilla policy gradient empirically? In this paper, we present a different perspective on deep policy gradient methods within the actor-critic framework: accurate value estimation is more critical than enforcing trust regions. We first demonstrate that, in practice, trust region methods do not behave as their theoretical analysis suggests. Then, we analyze how TRPO and PPO implicitly enhance value estimation through their implementation designs. Our theoretical analysis reveals that the true value often grows significantly faster than the critic’s predictions, leading to poor value estimation during policy training. Finally, empirical results show that simply increasing the number of value updates enables the basic VPG algorithm to match PPO’s performance across multiple continuous control benchmarks in Gymnasium. This highlights the pivotal role of value estimation in improving policy gradient methods. This paper is organized as follows. In Section 4 , we analyze the relationship between trust regions and policy improvement, demonstrating that there is no clear correlation. This suggests that enhancing a trust region may not be the fundamental reason for the success of trust region methods. In Section 5 , we explain that both TRPO and PPO perform more value steps per iteration compared to VPG, fundamentally contributing to closer value approximation and improved performance. Additionally, we provide a theoretical framework suggesting that value networks typically require more gradient steps to optimize than policy networks. In Section 6 , extensive experiments are conducted and presented to corroborate our theoretical analysis. Specifically, we show that by increasing the number of value steps alone, VPG achieves performance similar to or better than PPO across several Gymnasium benchmarks. 2 Related Work Performance gap between VPG and PPO. Much work has been done to understand the performance gap between VPG and PPO. It has been shown that the VPG loss performs significantly worse than the PPO loss, even with optimally conditioned hyperparameters ( Andrychowicz et al., 2021 ; Raffin et al., 2021 ) . On the theoretical side, the global optimality of TRPO/PPO is proven for overparameterized neural networks under the assumption of a finite action space ( Liu et al., 2019 ) . It is also shown that ratio clipping may not be necessary in PPO ( Sun et al., 2023 ) . Another theory suggests that policy gradients work by smoothing the value landscape ( Wang et al., 2024 ) , indicating that the choice of policy objective may not be a critical factor affecting performance, as all of them provide the same smoothing effect through Gaussian kernels. In this work, we take a step forward to demonstrate that the fundamental gap between VPG and PPO lies in value estimation, and by optimizing it, VPG can achieve performance comparable to PPO. Value estimation in off-policy methods. Although it has been demonstrated that an optimal baseline in policy gradients can effectively reduce the variance of the policy gradient estimator ( Peters & Schaal, 2008 ) , it remains unclear to what extent value estimation affects the performance of on-policy algorithms. In contrast, the importance of value estimation has been extensively studied in off-policy algorithms. In particular, it has been found that the \max operator may lead to overestimated Q values, and double Q learning methods were introduced to mitigate this error in estimation ( van Hasselt, 2010 ; van Hasselt et al., 2016 ) . This method also serves as one of the keystones in advanced off-policy algorithms such as TD3 ( Fujimoto et al., 2018 ) and SAC ( Haarnoja et al., 2018 ) , compared to the DDPG algorithm ( Lillicrap et al., 2015 ) . Additionally, it has been numerically shown that using state-action-dependent baselines does not reduce variance compared to state-dependent baselines in benchmark environments ( Tucker et al., 2018 ) . Implementation matters in deep policy gradients. Despite its accomplishments, deep RL methods are notorious for their brittleness, lack of stability and reliability ( Henderson et al., 2018 ; Engstrom et al., 2020 ) . Furthermore, it has been reported

Read the complete text