Analysis of On-Policy Policy Gradient Methods Under Distribution Mismatch
Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan corresponding · 2025
Read Analysis of On-Policy Policy Gradient Methods Under Distribution Mismatch by Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan corresponding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22244.txt.
Introduction
Weizhen Wang and three co-authors analyze a quiet flaw in policy gradient learning: the sampling distribution does not match the theory. Their work appeared in twenty twenty-five. These methods train decision policies in reinforcement learning, and state-of-the-art versions sample states the wrong way.
For simple table-based policies, the authors prove global convergence despite the biased updates. For the softmax form, they compute the exact leading error terms under the bias. Their ratio tells when the mismatch speeds learning and when it slows it. Bounds for general policies show both mismatches shrink at least linearly as the discount nears one. The work explains why a theoretically untidy method works so well in practice.
Catalog record
| CREATOR | Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan corresponding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analysis of On-Policy Policy Gradient Methods Under Distribution Mismatch Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan corresponding Abstract Policy gradient methods are among the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical success, many state-of-the-art policy gradient algorithms for discounted problems estimate update directions using a sampling distribution that differs from the discounted state distribution required by the policy gradient theorem. In this work, we analyze the impact of this distribution mismatch on policy gradient methods. Specifically, for tabular direct and softmax parameterizations, we establish global convergence of objective values under biased updates. For tabular softmax policies, we further derive the exact leading-order coefficients of the asymptotic objective errors under biased and normalized exact updates. Their ratio provides an explicit criterion for determining when distribution mismatch accelerates or slows asymptotic convergence. We then extend the analysis to more general policy parameterizations by deriving explicit bounds on both the state distribution mismatch and the induced normalized policy-gradient mismatch in episodic and continuing MDPs. These mismatches vanish at least linearly as the discount factor approaches one. Under an additional smoothness assumption, we further show that the limit inferior of the squared norm of the normalized exact policy-gradient direction along the biased iterates is bounded by O ( ( 1 − γ ) 2 ) O((1-\gamma)^{2}) in both settings. Our results provide theoretical insights into the robustness of policy gradient methods to distribution mismatch and shed light on the gap between their theoretical formulations and practical implementations. keywords Convergence analysis, distribution mismatch, policy gradient methods, reinforcement learning. † † thanks: [ † † address: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China , , , corresponding]Corresponding author: Xiaoming Duan. 1 Introduction 1.1 Problem Description and Motivation Policy gradient algorithms aim to identify a parameterized policy that optimizes a performance metric by iteratively updating the policy parameters along a gradient direction. In reinforcement learning (RL), popular policy gradient methods, such as Trust Region Policy Optimization (TRPO) ( Schulman et al., 2015 ) , Asynchronous Advantage Actor-Critic (A3C) ( Mnih et al., 2016 ) , and Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) are widely used to maximize the expected discounted cumulative reward and have demonstrated strong empirical performance. Since the policy gradient can be expressed as an expectation with respect to a state distribution, it is commonly estimated from trajectory samples in model-free implementations. However, as noted in ( Thomas, 2014 ) , there exists a discrepancy between the practical sampling distribution and the discounted state distribution required by the policy gradient theorem . Specifically, many on-policy implementations collect state-action pairs by executing the current policy, store them in a rollout buffer, and estimate the update direction using minibatches sampled uniformly from the buffer . Since the theoretically required state distribution weights state visits according to their temporal discount factors , uniform sampling from the rollout buffer without corresponding weighting or correction leads to a distribution mismatch. This distribution mismatch introduces bias into gradient estimates, resulting in biased update directions in many widely used policy gradient algorithms. Consequently, standard guarantees for exact gradient ascent do not directly apply, and such biased updates can lead to suboptimal performance in certain settings ( Nota and Thomas, 2020 ) . Nevertheless, due to their ease of implementation and strong empirical performance, biased policy gradient algorithms remain widely adopted, often without explicitly correcting the distribution mismatch . Understanding the effect of this mismatch and why such algorithms can remain effective therefore calls for further theoretical analysis. . 1.2 Related work The state distribution mismatch in policy gradient methods has been highlighted in numerous studies ( Thomas, 2014 ; Nota and Thomas, 2020 ; Zhang et al., 2022 ; Wu et al., 2022 ; Tosatto et al., 2022 ; Che et al., 2023 ; Pan et al., 2023 ) . Existing studies have investigated both its effects on policy optimization and methods for correcting the resulting bias. Zhang et al. ( 2022 ) consider this mismatch in the episodic setting. Using the temporal policy-gradient formulation in ( Nota and Thomas, 2020 ) , they implement DisPPO by restoring the factor \gamma^{t} in the actor update for each transition observed at time t within an episode. This approach requires tracking the within-episode time index and assigns exponentially smaller weights to transitions occurring later in an episode . Their experiments show that the uncorrected PPO can outperform DisPPO in some tasks, and they interpret this advantage through the role of later transitions in representation learning using an auxiliary-task formulation. Since the bias originates from a distribution shift, another common approach is to correct the mismatch through importance sampling. Che et al. ( 2023 ) focus on the continuing setting and introduce an averaging correction factor that theoretically corresponds to the importance ratio. To approximate this ratio in practice, they collect timestamped transitions in a rollout buffer and train an auxiliary neural network by regression . This state-dependent correction provides a lower-variance alternative to directly weighting samples by \gamma^{t} , but requires learning an additional correction model. Instead of correcting the distribution, Tosatto et al. ( 2022 ) derive an alternative formulation of policy gradient that avoids the discounted distribution by connecting it with the gradient of the action value function \nabla_{\theta}Q^{\pi} , which they refer to as the gradient critic. If both the action value function Q^{\pi} and the gradient critic \nabla_{\theta}Q^{\pi} are known, then an exact policy gradient can be computed by averaging over the initial state distribution, thereby avoiding reliance on the discounted state distribution. They further show that the gradient critic satisfies a Bellman equation, allowing it to be estimated through temporal-difference methods. This approach avoids explicit distribution correction but requires estimating the gradient critic in addition to the action-value function. Pan et al. ( 2023 ) investigate several heuristic bias rectification approaches, including learning rates adjustment, adaptive optimizers, and KL regularization , through extensive experiments, but their study is primarily empirical. These studies provide methods for correcting the mismatch and insights into the empirical effectiveness of biased updates. Complementing these efforts, we focus on the optimization guarantees that remain available without distribution correction. Specifically, we study whether biased updates preserve global convergence under tabular parameterizations, compare the exact asymptotic convergence rates of biased and exact updates under tabular softmax parameterization, and how the mismatch affects stationarity guarantees under more general parameterizations. 1.3 Contributions Our main contributions are summarized as follows: • Convergence and exact asymptotic rates under tabular parameterizations. We establish global objective-value convergence under biased updates for both direct and tabular softmax parameterizations. For tabular softmax policies under a unique optimal action assumption, we further derive the exact leading-order coefficients of the asymptotic objective errors for the biased and exact updates. Their ratio provides an explicit comparison of their asymptotic convergence speeds. • Distribution and gradient mismatch bounds for general parameterizations. For episodic and continuing MDPs, we derive explicit bounds on the mismatch between the discounted and undiscounted state distributions. Under general policy parameterizations with bounded policy derivatives, we further bound the induced normalized policy-gradient mismatch and show that