CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning
Tianshi Xu, Yuteng Chen, Meng Li · 2026
Read CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning by Tianshi Xu, Yuteng Chen, Meng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15141.txt.
Introduction
Tianshi Xu, Yuteng Chen, and Meng Li clean up the training trails of tool-using language models. Their 2026 arXiv paper deals with agentic reinforcement learning. Models learn by acting, often calling tools such as a Python interpreter to solve hard problems.
For small models, the exploration phase is messy. Runs fail often, and those noisy trails hinder learning. The authors present CLEANER, a method based on self-purified trajectories. By cleaning its own trails first, the model gives the whole training process a much clearer and cleaner signal for the learner to learn from.
Catalog record
| CREATOR | Tianshi Xu, Yuteng Chen, Meng Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15141
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning Tianshi Xu, Yuteng Chen, Meng Li Abstract Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B–7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue , where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking , while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER . Distinct from external filtering methods, CLEANER exploits the model’s intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6 %, 3 %, and 5 % over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub . Figure 1: Left : Illustration of the differences between the standard baseline and our CLEANER. Right : By reducing the number of tool execution failures within trajectories during training, our method improves pass@1 accuracy on AIME’25 by 8.1%. 1 Introduction The landscape of Large Language Models (LLMs) is shifting from passive text generation systems toward autonomous agents that solve complex tasks through tool use ( 46 ; 11 ; 30 ; 16 ; 20 ; 9 ) . Among the diverse tool modalities available to LLM agents, the Python code interpreter plays a particularly critical role 40 ; 31 ; 48 . Due to its Turing completeness and deterministic execution semantics, Python is indispensable for tasks that require precise computation, including mathematical reasoning, algorithmic problem-solving, and data analysis. As observed by Ronacher ( 29 ) , code is increasingly serving as a “universal interface” that unifies logical reasoning, computation, and API interaction within a single expressive medium. This perspective aligns with frameworks like CodeAct ( 40 ) , which advocate for treating code execution as a first-class action in agentic reasoning. By doing so, agents can effectively plan, verify intermediate results, and iteratively correct errors through interaction with an execution environment. Together, these insights highlight that robust code synthesis and execution are foundational capabilities for tool-augmented LLM agents. Motivated by this perspective, this work focuses on Python code execution as the primary tool modality. However, fully realizing this potential presents significant challenges for parameter-constrained models (e.g., 4B–7B). A primary obstacle is the high rate of execution failure, particularly during the exploration phase of Reinforcement Learning (RL) ( 12 ) . Before policy convergence, these models frequently generate invalid code, causing the intended recovery mechanism to degenerate into prolonged “Error \rightarrow Feedback \rightarrow Retry” loops, as depicted in Figure 1 (left). This instability constitutes a critical bottleneck in training. As evidenced by the experimental results in Figure 1 (right), an excessive accumulation of tool errors within trajectories closely correlates with performance bottlenecks or even accuracy degradation. We attribute this to the pollution of context: repeated failures generate large amounts of misleading signals (e.g., invalid code and verbose tracebacks). This accumulated noise likely causes semantic interference, biasing the model toward rationalizing incorrect execution paths rather than re-grounding its decisions, thereby hindering policy improvement. In principle, RL algorithms are expected to guide models away from such instability ( 47 ; 3 ) . However, standard training paradigms frequently worsen the problem due to the credit assignment issue . Under sparse, outcome-based reward settings like GRPO 12 , the entire trajectory receives a uniform positive reward upon final success, regardless of preceding failures. This mechanism fails to distinguish between efficient reasoning and trajectories containing errors, effectively treating them as equivalent. Consequently, erroneous tool usage and the underlying logic are inadvertently reinforced despite their negative impact on reasoning. To mitigate this, prior research has explored various strategies, yet each introduces new flaws. Attempts to assign dense rewards for individual tool executions often suffer from reward hacking , biasing agents toward optimizing intermediate metrics rather than final outcomes 48 . Alternatively, works such as rstar2-agent ( 31 ) utilize supersampling-based trajectory filtering, retaining only high-quality instances from 2\times generated candidates. However, this incurs a prohibitive computational cost. Since the rollout phase usually dominates RL training (accounting for >80\% of runtime ( 21 ; 32 ; 10 ) ), such extensive sampling renders these strategies unscalable for resource-constrained settings. To address these challenges, we propose CLEANER ( Self-Purified Trajectories Boost Agentic Reinforcement Learning ). CLEANER significantly boosts the agentic RL by eliminating error-contaminated context from the training data. Unlike methods that rely on increasing rollout multiplicity, CLEANER operates specifically at the data level to refine the trajectories used for policy optimization. At the core of our approach is the Similarity-Aware Adaptive Rollback (SAAR) mechanism, which constructs self-purified trajectories. When the model generates incorrect code but subsequently self-corrects within the same rollout, SAAR intervenes to prevent the error-laden history from being used for optimization. Instead, it applies a retrospective context substitution where the trajectory is rolled back to the failure point and the erroneous action is replaced with the corrected solution. This process yields a revised trajectory containing substantially fewer execution errors. To ensure semantic coherence, SAAR adaptively regulates the rollback granularity based on the semantic similarity between the erroneous code and its corrected counterpart. High-similarity cases typically correspond to minor execution errors and trigger a shallow replacement that preserves the original reasoning. Conversely, low-similarity cases signal deeper logical flaws and necessitate the substitution of the entire reasoning segment to maintain consistency. By leveraging these self-purified trajectories, CLEANER reduces noise in the learning signal and accelerates capability acquisition. Empirical evaluations show that CLEANER outperforms standard baselines with average accuracy gains of approximately 6% on AIME, 3% on GPQA 28 , and 5% on LiveCodeBench 15 . Furthermore, it matches the performance of state-of-the-art (SOTA) models 48 while requiring only one-third of the RL steps. In summary, our main contributions are as follows: ❶ We propose CLEANER , which resolves the credit assignment dilemma in agentic RL by training on self-purified trajectories . This approach enables models to directly internalize correct reasoning patterns while filtering out the interference of execution noise. ❷ We introduce the SAAR mechanism to autonomously construct these clean signals. SAAR adaptively repairs failures—ranging from minor syntax typos to deep logical flaws—without the computational overhead of supersampling. ❸ We demonstrate that CLEANER achieves state-of-the-art efficiency and performance. It outperforms baselines with accuracy gains of 6% on AIME and 5% on LiveCodeBench, and notably matches SOTA performance using only one-third of the training steps. ❹ We provide a fully reproducible training pipeline and have made our code, environment configurations, and processed datasets available via GitHub to support further research. 2 Preliminaries 2.1 Agentic Reasoning