A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
Shiye Lei, Zhihao Cheng, Dacheng Tao · 2026
Read A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization by Shiye Lei, Zhihao Cheng, Dacheng Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22718.txt.
Introduction
Shiye Lei, Zhihao Cheng, and Dacheng Tao steady policy training for reasoning models by weighing the prefix of each try. RL post-training coaxes reasoning from models, but rollouts come from an older policy than the one being updated. Their 2026 preprint notes that this off-policy gap can shake training.
Their prefix importance ratio reweights each rollout by its early tokens. A steadier update should keep gains and cut collapses. The work targets the costly heart of modern model training. Small stability wins there pay off at scale. It is a focused fix, well placed.
Catalog record
| CREATOR | Shiye Lei, Zhihao Cheng, Dacheng Tao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization Shiye Lei, Zhihao Cheng, Dacheng Tao Abstract Reinforcement learning (RL) post-training has increasingly demonstrated strong ability to elicit reasoning behaviors in large language models (LLMs). For training efficiency, rollouts are typically generated in an off-policy manner using an older sampling policy and then used to update the current target policy. To correct the resulting discrepancy between the sampling and target policies, most existing RL objectives rely on a token-level importance sampling ratio, primarily due to its computational simplicity and numerical stability. However, we observe that token-level correction often leads to unstable training dynamics when the degree of off-policyness is large. In this paper, we revisit LLM policy optimization under off-policy conditions and show that the theoretically rigorous correction term is the prefix importance ratio, and that relaxing it to a token-level approximation can induce instability in RL post-training. To stabilize LLM optimization under large off-policy drift, we propose a simple yet effective objective, Minimum Prefix Ratio ( MinPRO ). MinPRO replaces the unstable cumulative prefix ratio with a non-cumulative surrogate based on the minimum token-level ratio observed in the preceding prefix. Extensive experiments on both dense and mixture-of-experts LLMs, across multiple mathematical reasoning benchmarks, demonstrate that MinPRO substantially improves training stability and peak performance in off-policy regimes. \DeclareCaptionType listing[Listing][List of Listings] A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization † † * Work done during an internship at ByteDance BandAI . † † \dagger Corresponding authors: [email protected] , [email protected] (a) An overview of MinPRO (b) AIME24 scores under off-policy training Figure 1: (a) An overview of MinPRO. \nabla_{\theta}\mathcal{J} denotes the policy gradient and \rho_{t} is the token-level importance sampling (IS) ratio. We take a step back to derive the full IS ratio \rho_{1:t} , referred to as the prefix importance ratio. MinPRO is then developed by relaxing \rho_{1:t}=\rho_{1:t-1}\cdot\rho_{t} to a non-cumulative proxy \underline{\rho}_{t}\rho_{t} . (b) AIME24 and AIME25 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy training. 1 Introduction Post-training has become an essential stage in modern large language model (LLM) development, complementing base model pre-training by aligning model behavior with human objectives and improving reasoning quality [ 11 , 6 ] . In many practical scenarios, only final ground truth answers are available, and collecting detailed chain-of-thought annotations is expensive and often impractical. Reinforcement learning (RL) has therefore emerged as the most effective and widely-used framework for LLM post-training [ 5 ] . When optimizing LLMs, an off-policy workflow is typically adopted: rollouts are generated by an older behavior policy \pi_{\mathrm{old}} , while the optimization target is a newer policy \pi . This design is primarily driven by system efficiency considerations. In practice, rollouts are often generated in large batches, which helps alleviate GPU idle time caused by response-length imbalance and improves infrastructure throughput [ 7 , 4 ] . These rollout batches are then divided into multiple mini-batches for gradient updates. Moreover, recent asynchronous RL frameworks further decouple rollout generation from policy gradient updates, introducing additional policy lag and amplifying off-policyness [ 10 , 12 ] . Although operating in a large off-policy regime can substantially improve rollout efficiency, the resulting rollout distribution shift poses severe challenges to optimization stability and often leads to training collapse under high off-policyness [ 23 , 29 ] . To correct the rollout distribution discrepancy arising in off-policy training, a standard approach is to apply the importance sampling ratio \pi/\pi_{\mathrm{old}} . Following the seminal PPO formulation [ 13 ] , most LLM RL objectives adopt a token-level importance sampling ratio due to its computational simplicity and numerical stability. However, by revisiting the policy gradient under off-policy conditions, we show that the theoretically rigorous correction term is the prefix importance ratio rather than the token-level ratio. While the token-level relaxation is often effective in near on-policy settings, it fails to capture the true policy mismatch in LLMs given the long rollouts typically generated. As a result, this approximation frequently leads to unstable optimization and inferior or even collapsed training under large off-policy drift. Motivated by this insight, we seek to incorporate prefix-level information to stabilize policy optimization. In autoregressive LLMs, the prefix importance ratio is defined as a cumulative product, which is often numerically unstable and therefore impractical to use directly. To address this challenge, we introduce Minimum Prefix Ratio ( MinPRO ), a simple yet effective objective to stabilize LLM post-training. MinPRO replaces the cumulative prefix ratio with a simple surrogate: the current token ratio multiplied by the minimum token ratio observed in the preceding prefix (see Figure 1(a) ). This formulation preserves essential prefix-level information while mitigating numerical instability and length bias by avoiding explicit cumulative products. To comprehensively evaluate the stability of MinPRO, we conduct RL post-training under a large off-policy regime and compare it against several widely used and strong baselines, including GRPO [ 14 , 26 ] , GSPO [ 28 ] , CISPO [ 1 ] , and M2PO [ 29 ] . As pre-trained models, we consider two dense LLMs, Qwen3-8B-Base and Qwen3-14B-Base, as well as the mixture-of-experts (MoE) model Qwen3-30B-A3B-Base [ 24 ] . We assess performance on a suite of mathematical reasoning benchmarks, including AMC23, AIME24, AIME25, MATH500, Olympiad, Minerva, and GSM8K. From the training reward curves, which play a role analogous to training loss curves in supervised learning, MinPRO consistently exhibits higher rewards and more stable optimization dynamics throughout training. Evaluation on downstream benchmarks further shows that MinPRO achieves the highest peak performance among all compared methods, demonstrating strong stability under off-policy conditions. 2 Related Works LLM Policy Optimization Pre-training equips LLMs with broad factual and linguistic knowledge, yet additional techniques are required to strengthen their reasoning ability. Chain-of-thought (CoT) prompting, which encourages models to decompose problems into intermediate steps, has emerged as an effective approach for improving reasoning performance [ 22 ] . More recently, RL-based post-training has been shown to induce CoT behaviors without requiring explicit step-by-step annotations. Seminal systems such as OpenAI-o1 [ 6 ] and DeepSeek-R1 [ 5 ] leverage RL to incentivize deliberate multi-step reasoning, leading to substantial gains on challenging tasks including mathematical problem solving and code generation. Many RL objectives have been developed from different perspectives, including clip-range design [ 26 , 25 , 17 ] , token ratio [ 20 ] , and token entropy [ 2 , 21 , 9 ] . In contrast to hard clipping, which discards all tokens whose importance ratios fall outside the trust region, CISPO [ 1 ] introduces a soft-clipping mechanism that preserves all token-level gradients while constraining only the magnitude of the importance-weighting factor. In this paper, we develop an RL objective grounded in the prefix importance ratio. While GSPO [ 28 ] replaces the token-level ratio with a full-sequence importance ratio, our approach neither discards token-level information nor relies on the complete sequence ratio. Instead, we focus on the prefix ratio, which provides a more flexible and fine-grained correction signal. Off-Policy RL in LLM Off-policy training, where rollouts are sampled using an older policy version and then used to update a newer policy, is common in LLM post-training. Using stale rollouts brings notable efficiency benefits. First, generating a large batch of rollouts at once