Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin · 2026

Read Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification by Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21244.txt.

Introduction

Yiju Guo, Tianyi Hu, Zexu Sun, and Yankai Lin look at why training language models with reward checks often wastes effort. The method works well for reasoning, but it explores poorly when try-outs are limited. That leads to few successes and shaky training on hard tasks.

The team finds that many failures do not come from hard problems. A small number of prompt tokens create noise that blocks the search. Cleaning those tokens out of the instructions lets the model use its attempts far better. The work points to prompt quality, not just model size, as a key limit on reasoning gains.

Catalog record

CREATORYiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21244

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework ( Lens ) , which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that Lens significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6 × \times speedup. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research. † † footnotemark: † † footnotetext: 🖂 Corresponding author: Yankai Lin. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR), such as GRPO ( 25 ) , has significantly advanced the reasoning capabilities of large language models (LLMs). However, RLVR fundamentally relies on sampling correct rollouts to generate informative learning signals ( 35 ; 39 ) . In complex reasoning tasks, reward sparsity arises from long-horizon decision making with delayed and binary feedback. When combined with a high-dimensional action space, correct rollouts become exceedingly rare (Figure 2 a), leading to a lack of positive samples and causing training to collapse or become highly inefficient 12 . Figure 1: An example of interference token purification : Removing a few interference tokens corrects the reasoning rollout and turns it into a successful one. Figure 2: (a) Zero-Reward Prompt Analysis: Comparison of the zero-reward prompt ratio across different models and rollout sizes ( n ). Lens significantly reduces the proportion of zero-reward samples compared to GRPO, enhancing training efficiency. (b) Distribution of token-level Interference Scores (log scale): Only a few tokens exhibit high interference. (c) Rollout Accuracy Improvement: Removing these interference tokens leads to an improvement in rollout success rates (Average@8). To mitigate these issues, recent works have primarily followed two directions: (1) scaling exploration by increasing rollout ( 31 ; 34 ; 37 ; 30 ) , and (2) filtering zero-variance prompts ( 35 ; 39 ) . However, the former has substantially higher computational cost without improving efficiency, while the latter sacrifices exploration on challenging samples, limiting the model’s ability to solve complex problems. As a result, neither approach truly addresses the core issue of inefficient exploration on challenging samples. To address this, we investigate why the model fails to explore successful rollouts. Through a fine-grained token-level analysis, surprisingly, we find that many failures arise not from problem difficulty, but from a few ( <5\% ) tokens that introduce excessive interference , as shown in Figure 1 and Figure 2 b. We define tokens that are likely to cause failures as Interference Tokens . Simply pruning these tokens improves rollout accuracy on previously failed DeepMath ( 14 ) samples by over 20% across all the model families (Figure 2 c). Building on these insights, we introduce Le ss N oise S ampling Framework ( Lens ) , an online selective rollout framework that improves high-quality evolution by extracting informative learning signals from low-success prompts. In the first stage, Lens identifies and removes interference tokens within low-success prompts via interference score (Figure 2 b), producing prompts that yield a higher proportion of successful rollouts. In the second stage, Lens performs a transfer from the purification process to the original noisy setting: successful rollouts generated under denoised prompts are used as high-reward supervision to calibrate policy optimization on the original prompts. Unlike standard filtering, this mechanism encourages the model to learn to ignore interference tokens, rather than merely fitting solutions under cleaner conditions, ultimately enhancing the robustness of LLM reasoning through self-exploration. Experimental results show that Lens significantly outperforms GRPO, achieving a Pareto improvement in performance–efficiency trade-offs, with an average performance gain of 3.88% and over 1.6 \times faster convergence across seven math reasoning benchmarks. Furthermore, Lens exhibits superior performance over both scaling exploration and prompt filtering baselines while using substantially fewer computational resources. These results further provide empirical support for our hypothesis: low-success, challenging prompts contain valuable training signals, highlighting the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research. Figure 3: Method Overview. In the first stage, Lens identifies and purifies interference tokens within low-success prompts via Interference Score (Defined in Section 2.1 ), thereby generating a higher proportion of successful rollouts. In the second stage, Lens uses successful rollouts from the denoised prompts as high-reward supervision to calibrate policy optimization on the original prompt, correcting gradient updates distorted by interference. 2 Lens : Less Noise Sampling Framework We introduce Lens ( LE ss N oise S ampling Framework), a plug-and-play rollout framework designed to facilitate effective policy exploration. Lens consists of two key components: (1) Interference Token Identification and Purification : identifying and purifying interference tokens in low-success prompts through interference score; (2) Calibrated Rollout Policy Optimization (CRPO) , an efficient post-training algorithm that transfers learning signals from the denoised prompts produced by Component (1) to the original noisy prompts, equipping the model with the ability to ignore interference and perform robust reasoning under noisy inputs. 2.1 Interference Token Identification and Purification In this section, we first describe how interference tokens are identified and then explain how prompt purification is performed. Interference Token Identification. We start from the observation that the reference model \pi_{\text{ref}} provides a stable reference distribution learned from the training data. In contrast, large token-level deviations of the learned policy \pi_{\theta} from this reference often signal over-optimization or spurious behavior driven by noise, which can destabilize exploration 23 . Motivated by this intuition, we identify interference tokens by measuring the token-level deviation between the current policy and the reference model. Specifically, for a token prefix s and the next generated token a , we define Interference Score as: S_{I}(s,a)\triangleq\bigl|\log\pi_{\theta}(a\mid s)-\log\pi_{\text{ref}}(a\mid s)\bigr| (1) Tokens with large interference scores contribute disproportionately to the KL divergence from the reference distribution and are therefore treated as interference tokens. Such deviations are commonly induced by reward over-optimization or noisy and misleading signals 23 ; 8 , and can hinder effective exploration and generalization in the high-dimensional token action space 7 ; 6 . Interference Purification. Based on the observations in Section 1 and the Interference Score , we propose a token-wise inspection and pruning mechanism. By removing only a small number of tokens, this mechanism effectively eliminates interference while preserving the original semantics of the prompt with minimal impact. Specifically, for the i -th prompt x_{i} , which contains |x_{i}| tokens after tokenization, we compute token-level Interference Scores for all the tokens in the prompt and rank the tokens in descending order. Then, we introduce a deletion ratio \gamma and select the top k=\lceil\gamma\cdot|x_{i}|\rceil tokens to form an interference token set I_{i} . We define the denoised prompt as x^{\prime}_{i}=x_{i}\setminus I_{i}

Read the complete text