Gradient Extrapolation-Based Policy Optimization

Ismam Nur Swapnil, Aranya Saha, Tanvir Ahmed Khan, [1ex] Mohammad Ariful Haque, Ser-Nam Lim · 2026

Read Gradient Extrapolation-Based Policy Optimization by Ismam Nur Swapnil, Aranya Saha, Tanvir Ahmed Khan, [1ex] Mohammad Ariful Haque, Ser-Nam Lim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06755.txt.

Introduction

Ismam Nur Swapnil and five co-authors introduce PrePO, a method that cheapens reinforcement learning fine-tuning of language models by skipping predictable optimizer steps. Repeatedly updating a policy on the same rollout batch costs a backward pass per step. PrePO watches two real optimizer transitions, estimates a farther point on the same trajectory, jumps partway there, evaluates the true objective, and applies a corrective update.

Its cost stays fixed no matter how many virtual steps it approximates. Analysis gives error bounds for common optimizers and shows accurate endpoint estimates preserve normal convergence behavior. Experiments reach matched performance in fewer steps and less time, and the same trick also works in supervised fine-tuning.

Catalog record

CREATORIsmam Nur Swapnil, Aranya Saha, Tanvir Ahmed Khan, [1ex] Mohammad Ariful Haque, Ser-Nam Lim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Extrapolation-Based Policy Optimization Ismam Nur Swapnil, Aranya Saha, Tanvir Ahmed Khan, [1ex] Mohammad Ariful Haque, Ser-Nam Lim Abstract Reinforcement learning is widely used to improve the reasoning ability of large language models, especially when answers can be automatically checked. Standard GRPO-style training updates the model using only the current step, while full multi-step lookahead can give a better update direction but is too expensive because it needs many backward passes. We propose Gradient Extrapolation-Based Policy Optimization (GXPO) , a plug-compatible policy-update rule for GRPO-style reasoning RL. GXPO approximates a longer local lookahead using only three backward passes during an active phase. It reuses the same batch of rollouts, rewards, advantages, and GRPO loss, so it does not require new rollouts or reward computation at the lookahead points. GXPO takes two fast optimizer steps, measures how the gradients change, predicts a virtual K K -step lookahead point, moves the policy partway toward that point, and then applies a corrective update using the true gradient at the new position. When the lookahead signal becomes unstable, GXPO automatically switches back to standard single-pass GRPO. We also give a plain-gradient-descent surrogate analysis that explains when the extrapolation is exact and where its local errors come from. Across Qwen2.5 and Llama math-reasoning experiments, GXPO improves the average sampled pass@1 by + 1.65 \mathbf{+1.65} to + 5.00 \mathbf{+5.00} points over GRPO and by + 0.14 \mathbf{+0.14} to + 1.28 \mathbf{+1.28} points over the strongest SFPO setting, while keeping the active-phase cost fixed at three backward passes. It also achieves up to 4.00 × \mathbf{4.00\times} step speedup, 2.33 × \mathbf{2.33\times} wall-clock speedup, and 1.33 × \mathbf{1.33\times} backward-pass speedup in reaching GRPO’s peak accuracy. 1 Introduction Policy-gradient reinforcement learning underlies modern language-model alignment and reasoning ( Sutton et al. 1999 ; Williams 1992 ; Schulman et al. 2017 ) . In reinforcement learning with verifiable rewards (RLVR), models generate candidate solutions, receive verifiable rewards, and update the policy with first-order gradients ( Shao et al. 2024 ; Yu et al. 2025 ) . This setting is central to MATH and GSM8K-style reasoning benchmarks, where correctness can be checked automatically after long-form generation ( Hendrycks et al. 2021 ; Cobbe et al. 2021 ) . At LLM scale, each added backward pass multiplies training time and memory, putting update quality in tension with per-step cost during repeated post-training iterations. Figure 1 : Overall GXPO training framework. Each active step performs three backward passes: two probe gradients during gradient extrapolation and one corrective gradient g_{\text{slow}} at the repositioned point \tilde{\theta}^{t} . The slow correction is always applied at the current step. After the update, the z -score gate checks \|g_{\text{slow}}\| ; if Z>\tau , all subsequent steps permanently fall back to single-pass GRPO. Single-step methods such as PPO and GRPO avoid this cost but use only the current-policy gradient ( Schulman et al. 2017 ; Shao et al. 2024 ) . Explicit lookahead recovers trajectory information by paying for it: h -step policy mirror descent improves regularized policy iteration ( Protopapas and Barakat 2024 ) , tree-expansion policies reduce policy-gradient variance ( Dalal et al. 2025 ) , and online or adaptive planning uses learned-model rollouts for action selection ( Sikchi et al. 2021 ; Rosenberg et al. 2023 ) . These methods rely on planning, tree expansion, or auxiliary value/model estimates rather than the rollout batch already in hand, making them difficult to drop into standard GRPO-style reasoning pipelines without changing the data path. Recent LLM reasoning-RL work improves stability and efficiency through objectives, values, entropy control, filtering, rewards, recipes, or off-policy reuse ( Liu et al. 2025 ; Dai et al. 2025 ; Yue et al. 2025 ; Xiong et al. 2025 ; Cui et al. 2025 ; Wen et al. 2025 ; Fatemi et al. 2025 ; Shen et al. 2025 ; Mroueh 2025 ; Mroueh et al. 2025 ) , and through sample selection, down-sampling, selective rollouts, few-example training, test-time scaling, or RLHF/generation-system acceleration ( Chen et al. 2024 ; Xia et al. 2024 ; Ye et al. 2025 ; Li et al. 2025 ; Xu et al. 2026 ; Wang et al. 2025 ; Zheng et al. 2025 ; Muennighoff et al. 2025 ; Sheng et al. 2025 ; Kwon et al. 2023 ; He et al. 2025 ) . GXPO is orthogonal: it keeps the rollout batch, rewards, advantages, and GRPO loss unchanged, and only changes the policy update. This makes the update rule a narrow intervention, not a new reasoning-RL training recipe or reward pipeline component. Optimizer-side lookahead is closer to our setting. The Lookahead optimizer interleaves fast inner steps with a slow update ( Zhang et al. 2019 ; Zhou et al. 2021 ) , and SFPO ports this idea to policy optimization with K fast inner steps before a slow correction ( Wang et al. 2026 ) —paying K{+}1 backward passes per update. The question we address is whether a policy update can gain similar gradient-trajectory information to explicit lookahead methods without increasing the number of backward passes. We introduce Gradient Extrapolation-Based Policy Optimization (GXPO) , a GRPO-compatible update that approximates K local policy-gradient steps with three backward passes, independent of virtual depth K (see Figure 1 ). GXPO reuses the rollout batch, rewards, advantages, and regularization; estimates a per-coordinate retention ratio r_{i}=g_{1,i}/g_{0,i} from two probe gradients; moves toward the geometric displacement; and applies a corrective gradient at the repositioned policy, so the final step remains anchored to the true objective rather than the extrapolated prediction. A rolling z-score gate falls back to a single-pass update when the corrective-gradient norm becomes unstable during training. Our contributions are: • A GRPO-compatible update with three active-phase backward passes for any virtual lookahead depth K , using two probe gradients, geometric extrapolation, and one corrective gradient; • A fixed-batch implementation that reuses rollouts, rewards, advantages, loss, and regularization, with a gate that reverts to the base single-pass update when the local trajectory signal becomes unreliable; • A local quadratic surrogate analysis, plus benchmark, budget, ablation, and diagnostic evidence across two model families. 2 Method: Gradient Extrapolation-Based Policy Optimization GXPO replaces a single GRPO update with a three-step update, while reusing the same rollouts, rewards, advantages, and objective, so no additional data or reward computation is required. During training, it first takes two quick optimization steps using the base actor optimizer (AdamW in our experiments ( Loshchilov and Hutter 2019 ) ) and observes how the parameters change. It then uses this change to estimate the direction of the update, moves partway in that direction, and applies one final corrective step. The theory studies a simplified version to clarify this behavior, while the experiments evaluate the method under the chosen optimizer. Setup and notation. Let \theta\in\mathbb{R}^{d} parameterise the policy and let \mathcal{L}(\theta) be the GRPO loss. Write g(\theta)=\nabla_{\theta}\mathcal{L}(\theta) , H(\theta)=\nabla_{\theta}^{2}\mathcal{L}(\theta) , and g_{n}=g(\theta_{n}) . Let \eta>0 be the learning rate and K the virtual depth. 2.1 From Taylor Expansion to Geometric Scaling GXPO needs a cheap way to estimate how the gradient would change over several nearby steps. The local intuition is simple: if the loss curvature is approximately stable near \theta_{0} , then the gradient evolution can be approximated by a predictable local

Read the complete text