Not All Tokens are Needed: Token-Efficient Reinforcement Learning

Hejian Sang 1 * † Yuanda Xu 1 * Zhengze Zhou 1 * Ran He 1 * Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA · 2026

Read Not All Tokens are Needed: Token-Efficient Reinforcement Learning by Hejian Sang 1 * † Yuanda Xu 1 * Zhengze Zhou 1 * Ran He 1 * Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06619.txt.

Introduction

Sang and colleagues address scaling reinforcement learning to long chain-of-thought trajectories, where updating every token turns length into a hidden training tax. They introduce Not All Tokens Are Needed (NAT), updating the policy from a token subset while preserving the learning signal. Its core is an unbiased partial-token estimator with Horvitz-Thompson reweighting, plus Uniform Random Sampling and Random Prefix Cutting selectors that cut compute and memory without changing rewards or rollouts. On mathematical reasoning benchmarks, NAT matches GRPO using as few as 50% of tokens.

Catalog record

CREATORHejian Sang 1 * † Yuanda Xu 1 * Zhengze Zhou 1 * Ran He 1 * Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49694
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06619

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Tokens are Needed: Token-Efficient Reinforcement Learning Hejian Sang 1 * † Yuanda Xu 1 * Zhengze Zhou 1 * Ran He 1 * Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA Abstract Reinforcement learning (RL) has become a key driver of progress in large language models, but scaling RL to long chain-of-thought (COT) trajectories remains increasingly constrained by backpropagation over every generated token . Even with optimized rollout engines, full-token updates can consume a large fraction of total training cost, turning token length into a hidden tax on RL. We introduce Not All Tokens are Needed (NAT) , a unified framework that makes the token budget a first-class optimization primitive: NAT updates the policy using only a selected subset of generated tokens while preserving the learning signal of full-sequence RL. The core idea is an unbiased partial-token policy-gradient estimator via Horvitz–Thompson reweighting, which ensures statistically correct gradients despite subsampling. We instantiate NAT with two simple, plug-and-play token selection schemes: Uniform Random Sampling (URS) and Random Prefix Cutting (RPC) , both of which reduce forward/backward compute and memory without modifying the reward computation or rollout pipeline. Across mathematical reasoning benchmarks, NAT matches full-token GRPO performance while using as few as 50% of tokens, providing an efficient and orthogonal pathway to scaling RL beyond the limits imposed by long trajectories. From our experiments, RPC can save 18 % peak GPU memory and 29 % forward and backward RL training time for Qwen3-8B RL training. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author: Hejian Sang< [email protected] > Figure 1: Barplots of Qwen3-8B RL training metrics with 95% CIs across 5 runs for GRPO (vanilla GRPO), URS (GRPO with random sampling p=0.5 ), Det. Trunc. (GRPO with deterministic prefix truncation of 50% tokens) and RPC (GRPO with uniform random prefix cutting). 1 Introduction Recent generations of Large Language Models (LLMs) Ziegler et al. 2019 ; Ouyang et al. 2022 ; Stiennon et al. 2020 have demonstrated striking advances in mathematical reasoning Guo et al. 2025 , code synthesis Pan et al. 2024 , and scientific problem solving. A major driver behind these gains is post-training with reinforcement learning objectives that use verifiable signals (like unit tests for code, symbolic execution, or exact-answer checks for math) commonly referred to as Reinforcement Learning from Verifiable Rewards (RLVR) Jaech et al. 2024 ; Guo et al. 2025 ; Team et al. 2025 ; Yang et al. 2025 . By directly optimizing for task success, RLVR can push models beyond what supervised fine-tuning alone achieves, especially on benchmarks where solutions require multi-step chains-of-thought (CoT) and careful intermediate decisions. As a result, RLVR has become a core ingredient in modern reasoning systems, including OpenAI O1 Jaech et al. 2024 , DeepSeek-R1 Guo et al. 2025 , Kimi 1.5 Team et al. 2025 , and Qwen3 Yang et al. 2025 . Despite its effectiveness, RLVR remains expensive and increasingly difficult to scale. A central reason is that standard RLVR pipelines treat all generated tokens as equally important for learning: they compute policy-gradient losses and backpropagate through every token in each rollout trajectory. However, as RLVR succeeds at eliciting deeper reasoning, trajectories often become longer and more variable, which amplifies activation memory, increases per-update FLOPs, and can trigger out-of-memory (OOM) failures or unstable optimization dynamics Guo et al. 2025 ; Yeo et al. 2025 . In practice, the cost of RLVR grows not only with the number of rollouts, but also with the length of each rollout and long-CoT behavior can become the training bottleneck. A typical GRPO Shao et al. 2024 training cycle further highlights an efficiency mismatch across stages. Rollout generation is often the most visible bottleneck and can account for a large fraction of wall-clock time (e.g., around 70% in representative profiles Gao et al. 2025 ). Consequently, recent work has aggressively optimized generation through high-throughput inference engines ( Kwon et al. 2023 ; Zheng et al. 2024 ) , speculative decoding ( Leviathan et al. 2023 ) and low precision inference engine Xi et al. 2026 . Yet, even when generation is accelerated, the subsequent learning phase—forward and backward passes over full trajectories—remains memory-bound and compute-intensive. This creates a practical ceiling: we can generate rollouts faster, but we cannot consume them efficiently enough to translate that throughput into scalable policy improvement. This paper takes an orthogonal approach to RLVR efficiency. Instead of optimizing how rollouts are produced, we optimize how they are used by the learner. We ask a simple but consequential question: Do we really need all tokens to train strong RL reasoners? Intuitively, not all tokens in a long CoT contribute equally to the learning signal. Many tokens are “mechanical” continuation (for example, filling in algebraic steps) reiterating previously established context, or emitting low-entropy boilerplate, while a smaller subset corresponds to high-impact decision points that steer the trajectory. If we can update the policy using only a carefully chosen subset of tokens—without distorting the training objective, we can reduce memory and compute per optimizer update, increase training throughput, and unlock longer-context RLVR without sacrificing performance. We introduce Not All Tokens are Needed (NAT), a principled framework for token-efficient RLVR. NAT performs policy optimization using only a selected subset of tokens from each rollout while keeping reward signals computed on the full response. Concretely, NAT masks tokens during the policy-gradient computation, so that only included tokens contribute to backpropagation. This truncates the activation graph, reduces memory pressure, and decreases per-step compute. Importantly, NAT is not restricted to one masking rule: it can instantiate uniform token sampling, structured prefix-based schemes, or other selection mechanisms, as long as we can correct for selection bias appropriately. The core technical ingredient of NAT is Horvitz–Thompson (HT) estimation applied to token-masked policy gradients. Each token t is assigned a inclusion probability, and NAT reweights gradients by the inverse of that probability. We prove that this HT-corrected token-masked objective yields an unbiased estimator of the original full-sequence RLVR gradient. In other words, NAT preserves the target learning signal in expectation, even though each update only backpropagates through a subset of tokens. This provides a rigorous foundation for token-efficient RLVR: partial-token updates can remain statistically aligned with optimizing the full-sequence reward objective, rather than introducing systematic drift. We instantiate NAT on top of the GRPO objective and evaluate across multiple reasoning benchmarks and model scales. Empirically, NAT achieves performance comparable to full-token GRPO while using substantially fewer tokens in the backward pass (e.g., as low as 50% token coverage), translating into lower activation memory and higher training throughput. Moreover, NAT composes naturally with system-level optimizations (faster rollout engines, batching, scheduling), because it targets a different bottleneck: the cost of consuming long trajectories during learning. Contributions. Our main contributions are: • A unified framework for token-efficient RLVR. We propose Not All Tokens are Needed (NAT), which performs RLVR updates using masked token subsets while preserving full-response reward evaluation. NAT supports a broad family of token selection strategies, including uniform random token sampling and structured prefix-based masking. • Unbiasedness via Horvitz–Thompson correction. We show that HT reweighting yields an unbiased estimator of the full-token GRPO gradient for any positive inclusion probabilities. This establishes a principled connection between token masking and faithful

Read the complete text