Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
Dong Shu, Denghui Zhang, Jessica Hullman · 2026
Read Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training by Dong Shu, Denghui Zhang, Jessica Hullman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01597.txt.
Introduction
Shu and colleagues propose Influence-Guided PPO, a post-training method using data attribution. Standard Proximal Policy Optimization trains on the entire rollout buffer, assuming all episodes help. Episodes contain noisy or unfaithful reasoning, which can hurt model results and slow training. By scoring each episode's influence via gradient math, Influence-Guided PPO identifies and drops episodes anti-aligned with a gradient. Tests show Influence-Guided PPO beats supervised fine-tuning and PPO baselines. Filtering acts as an early stop, reducing unfaithful chain-of-thought reasoning.
Catalog record
| CREATOR | Dong Shu, Denghui Zhang, Jessica Hullman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77954 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01597
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training Dong Shu, Denghui Zhang, Jessica Hullman Abstract Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal. However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training. In this paper, we propose Influence-Guided PPO (I-PPO) , a novel framework that integrates data attribution into the RL post-training loop. By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient. Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines. We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency while effectively reducing unfaithful CoT reasoning. The code is available at https://anonymous.4open.science/r/Influence_ppo-4C37 . 1 Introduction Reinforcement learning (RL) has emerged as a standard technique for enhancing the performance and aligning the reasoning capabilities of Large Language Models (LLMs) ( 35 ; 42 ; 12 ) . Leading models, such as DeepSeek-R1 ( 11 ) , GPT-5 ( 33 ) , and Gemini-3 ( 37 ) , have demonstrated that while pre-training on vast corpora establishes a foundation of world knowledge, post-training via RL algorithms can further optimize reasoning capabilities beyond the limits of standard supervised fine-tuning ( 20 ; 31 ; 40 ) . However, traditional RL methods, such as PPO, are often computationally inefficient ( 16 ; 21 ) . Standard PPO trains the model on the entire rollout buffer generated by the current policy, operating under the assumption that every generated episode provides a beneficial signal for optimization ( 29 ; 30 ) . In practice, raw rollout buffers frequently contain low-quality or unfaithful samples, where the model arrives at the correct answer via flawed reasoning. These data introduce noise and degrade the policy update ( 46 ; 3 ; 39 ) . Moreover, the rollout buffer inevitably includes redundant episodes representing reasoning pathways the model has already mastered. Continuously optimizing the policy on these saturated knowledge provides negligible learning signal, further exacerbating computational inefficiency ( 46 ) . To filter out unfaithful and redundant episodes, we propose Influence-Guided PPO (I-PPO) , a novel framework that integrates local data attribution principles into the traditional PPO algorithm. Inspired by the TracIn data attribution function ( 25 ) , we quantify an episode’s influence by computing the gradient alignment between the generated episode in the rollout buffer and a validation set. The intuition is that in the context of LLM reasoning post-training, a beneficial episode should push the model’s parameters in the same direction as high-quality, human-preferred Chain-of-Thought (CoT) sequences. If an episode receives a negative influence score, it suggests that the episode may be pushing the model away from this target direction, and we filter it out prior to the policy update. This dynamic filtering process reduces the volume of the rollout buffer, thereby accelerating the training process while simultaneously allowing the model to achieve better overall performance. • We propose I-PPO, a novel framework that integrates data attribution to identify and eliminate episodes with negative influence before optimization. • Through experiments across mathematical, physical, and social reasoning domains, we demonstrate that our method outperforms SFT baseline and traditional PPO. • We show that our filtering process significantly accelerates training by dynamically reducing the rollout buffer volume as the model approaches convergence, effectively acting as an intrinsic early stopping mechanism. • We provide a fine-grained analysis revealing that I-PPO serves as an implicit, training-free reasoning reward signal, and effectively detects unfaithful episodes. Figure 1: Overview of I-PPO Framework. (a) Traditional PPO uses the raw rollout buffer for policy updates. (b) I-PPO refines the rollout by calculating influence scores for each generated episode. Negative episodes are removed. 2 Related Work Early data attribution methods, like Shapley values ( 9 ) , Influence Functions ( 18 ) and TracIn ( 25 ) , are designed to mathematically quantify exactly how much a specific data point or feature contributes to a model’s loss or parameter estimates. As model architectures scaled and training datasets grew, traditional exact verification methods, such as Leave-One-Out retraining ( 43 ) , became computationally impossible. Consequently, data attribution evolved into an approximate interpretability tool that can be used to trace model behaviors back to influential training examples. Early data attribution research such as 44 and 27 established foundational methods for quantifying the contribution of training data in deep learning architectures like CNNs and RNNs. Data attribution techniques are now increasingly applied to LLMs to support interpretability, addressing the challenges posed by the enormous scale of modern training corpora. To tackle the computational intractability of attribution on massive corpora, recent works like TrackStar ( 6 ) and LoGra ( 7 ) have developed efficient gradient approximation techniques specifically for the pretraining stage. Similarly, in the supervised fine-tuning stage, methods such as LESS ( 41 ) and TRAK ( 24 ) use low-rank gradient embeddings to identify influential instructions. However, a critical gap remains in the post-training reinforcement learning phase. While 16 introduced a local attribution framework for online RL and demonstrated its efficacy in scenarios like robotic control and toxicity detection, their approach primarily targets tasks with externally-defined environments and immediate reward signals. They do not address the unique internal dynamics of LLM reasoning, where the RL update does not involve an external environment, and the LLM may achieve a correct final answer through unfaithful or post-hoc reasoning chains. Our work demonstrates using data attribution as an implicit and approximate training-free process reward signal to detect and filter potentially spurious reasoning data that standard outcome-based rewards miss. 3 Preliminaries 3.1 PPO on Large Language Models PPO training optimizes a policy \pi_{\theta} against a reward signal while maintaining stability through clipping and KL regularization, which prevent the model parameters from deviating excessively from the previous policy. In this framework, policy \pi_{\theta} is the actor model, which is initialized from a SFT LLM. To prevent the model from deviating excessively from the distribution of natural language or forgetting its pre-training knowledge, a copy of the initial model is frozen and serves as the reference model, denoted as \pi_{\text{ref}} . The training process uses a reward model, R_{\phi}(x,y) , which estimates a scalar score for a prompt x and a generated response y . To ensure stability, the raw reward is modified by a Kullback-Leibler (KL) divergence penalty. The total reward function at time step t is defined as: r_{total}(x,y)=R_{\phi}(x,y)-\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right), (1) where \beta is the KL penalty coefficient controlling the strength of the regularization. As the policy \pi_{\theta} is being optimized, a critic model V_{\psi}(s_{t}) is also being trained to estimate the value function of the current state. In the context of LLMs, the state s_{t} encompasses the prompt x and all tokens generated up to step t . The critic model is initialized from the same LLM as the policy but with a scalar output head to predict the expected future return for each state. The critic model computes the Generalized Advantage Estimation (GAE), denoted as \hat{A}_{t}^{GAE(\gamma,\lambda)} ( 28 ) . Specifically, for each time