Not All Transitions Matter: Evidence from PPO Reducing Temporal Correlation in PPO Without Degrading Performance

Ajhesh Basnet [email protected] Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore · 2026

Read Not All Transitions Matter: Evidence from PPO Reducing Temporal Correlation in PPO Without Degrading Performance by Ajhesh Basnet [email protected] Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24071.txt.

Introduction

Basnet asks whether training can drop redundant transitions. In on-policy rollouts, each state follows from the last, so consecutive transitions overlap and the gradient signal repeats far more than the batch size suggests, destabilizing training. Randomly dropping a fixed share of transitions at the right stage breaks that repetition and steadies training with one sampling step and no change to PPO. Across five environments from CartPole to Hopper, it matches PPO on reward while steadying KL divergence, policy entropy, and value estimates, with dropping 25 percent the sweet spot.

Catalog record

CREATORAjhesh Basnet [email protected] Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33745
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24071

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Transitions Matter: Evidence from PPO Reducing Temporal Correlation in PPO Without Degrading Performance Ajhesh Basnet [email protected] Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore Abstract Training a reinforcement learning agent on-policy means collecting fresh experience at every update, and that experience comes with a hidden problem. Each state in a rollout is the direct output of the previous one, causally chained together by the agent’s own actions. Because of this, consecutive transitions are never truly independent. They carry overlapping information, and the gradient signal the network receives ends up far more repetitive than the batch size suggests. The same directions get reinforced over and over, the value network struggles to keep up as the policy shifts, and training becomes quietly unstable in ways that reward curves alone rarely reveal. This paper asks whether that redundancy can simply be removed. We show that randomly dropping a fixed fraction of transitions from the rollout, at the right stage so the reward signal stays intact, is enough to break the repetitive gradient structure and stabilise training. The change is minimal: one sampling step, no new components, no modification to the core algorithm, and it works with any PPO implementation. Across five environments of increasing difficulty, CartPole-v1, Acrobot-v1, LunarLander-v2, HalfCheetah-v5, and Hopper-v5, the method matches vanilla PPO on reward while producing more consistent training dynamics across KL divergence, policy entropy, and value estimates. Dropping 25 % 25\% of transitions turns out to be the sweet spot: enough to disrupt the redundancy, not enough to thin the batch. Source code: https://github.com/ajheshbasnet/rollout-slim . 1 Introduction Supervised learning assumes training samples are Independently and Identically Distributed (IID). Under this assumption, gradient signals across batches stay diverse and non-redundant, and learning proceeds stably. In on-policy RL, this assumption does not just get bent. It gets broken by construction. Within a single trajectory \tau , every state s_{t+1} is a direct causal product of s_{t} . The trajectory is not a collection of independent observations but a chain, where each transition follows from the policy’s own prior decisions. Feeding this into a neural network produces gradient vectors that point in nearly the same direction update after update, creating functionally collinear weight changes that slow and destabilize learning in ways that are hard to detect until training already looks wrong. There is a second problem that is harder to see. Once the policy updates past \tau_{1} , the value network starts operating on states it was never trained on. Stale value estimates feed into corrupted advantage signals, which feed into corrupted policy gradients, which push the agent into new regions of state space the critic has seen even less of, and the cycle keeps tightening. This is a direct instance of the Deadly Triad [ 2 ] : function approximation combined with bootstrapping under a non-stationary data distribution. Off-policy methods like DQN and SAC sidestep this through experience replay. Transitions get stored, shuffled, and randomly sampled, which approximates IID well enough in practice. On-policy methods like PPO [ 1 ] cannot do this. The data must come from the current policy, so old transitions cannot be reused, and the temporal correlation problem is left sitting there unaddressed in every rollout. What makes this worth investigating further is a counterintuitive observation: if you randomly drop a large fraction of transitions from the rollout and still match the performance of the agent that trained on all of them, it strongly implies that most of those transitions were carrying nearly the same gradient signal anyway. The states are so correlated with each other that the network was not really learning from each one independently — it was seeing the same information repeated. That is the core finding this paper builds on. What about vectorized environments? A common practical response to temporal correlation is to run 6–8 parallel environment copies simultaneously, collecting transitions from multiple independent trajectories at once. This does reduce correlation across the batch because each worker is at a different point in a different episode. It works. But running N environments in parallel means N times the memory and N times the CPU overhead, and on a single machine or constrained hardware that cost adds up fast. The method presented here achieves similar decorrelation benefits from a single environment rollout by subsampling after advantage estimation. It is not a replacement for vectorized environments in every setting, but it is a significantly cheaper path to the same goal. This paper looks at three methods for reducing that correlation without touching the core PPO objective or hurting performance. 2 Background 2.1 On-Policy Trajectory Collection In on-policy RL, the agent collects a trajectory by running its current policy \pi_{\theta} in the environment: \tau=(s_{0},a_{0},r_{0})\rightarrow(s_{1},a_{1},r_{1})\rightarrow(s_{2},a_{2},r_{2})\rightarrow\cdots\rightarrow(s_{T},a_{T},r_{T}) (1) Every transition (s_{t},a_{t},r_{t},s_{t+1}) is causally produced by the one before it, making the trajectory a temporally dependent chain rather than a collection of independent observations. 2.2 Temporal Correlation and Its Effect on Gradients The policy gradient objective is: \nabla J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t}\nabla\log\pi_{\theta}(a_{t}|s_{t})\cdot A_{t}\right] (2) This expectation assumes samples come from a stationary distribution. In practice, consecutive states s_{t} and s_{t+1} are highly similar, so gradient vectors \nabla\log\pi_{\theta}(a_{t}|s_{t}) and \nabla\log\pi_{\theta}(a_{t+1}|s_{t+1}) end up nearly parallel. 2.3 The Non-Stationary Bootstrapping Feedback Loop The TD update for the value network is: V(s_{t})\leftarrow V(s_{t})+\alpha\left[r_{t}+\gamma V(s_{t+1})-V(s_{t})\right] (3) Both V(s_{t}) and V(s_{t+1}) come from the same network whose weights are shifting during training. After training on \tau_{1} , the critic fits those states well and the loss is low. After n more trajectory updates the weights have drifted substantially. Evaluating the critic on \tau_{1} now gives high error, even though that error was low at the time of original training. This compounds across trajectories and gives rise to the Non-Stationary Bootstrapping Feedback Loop, a specific case of the Deadly Triad [ 2 ] : function approximation plus bootstrapping plus a shifting data distribution. To see why this is a problem, consider a simple analogy. Imagine a teacher who trains a student exclusively on calculus. Every example, every drill, every test comes from that one chapter. The student performs well on calculus. Then weeks later the teacher starts asking questions from algebra, assuming the student is equally well-prepared. The student struggles, not because they forgot calculus, but because they were never trained on the new material. The value network faces exactly this situation. It is trained on the state distribution of \tau_{1} , but as the policy updates the agent visits different states, and the distribution the critic is asked to evaluate shifts underneath it. The critic’s predictions become unreliable not because the network forgot what it learned, but because what it learned no longer matches what it is being asked about. 2.4 Why On-Policy Methods Must Discard Old Data Reusing trajectories from a past policy \pi_{\theta_{\text{old}}} to update the current policy \pi_{\theta} introduces distribution mismatch. The training data no longer reflects where the current policy actually goes, which biases the gradient in ways that compound over updates. Old trajectories are discarded not as a practical inconvenience but because using them violates the assumptions the objective was derived under. 2.5 Related Work The data efficiency and correlation problems in on-policy RL have attracted a growing body of work, each approaching the issue

Read the complete text