Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron · 2026

Read Scaling Reasoning Efficiently via Relaxed On-Policy Distillation by Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11137.txt.

Introduction

Jongwoo Ko and coauthors stabilize on-policy distillation. It is the route for moving reasoning skill from teacher models into students. But it can wobble or transfer negatively. They show the process acts like policy optimization. The teacher-student log-likelihood ratio serves as a token-level reward. They build REOPOLD on that view. It uses mixture-based reward clipping, entropy-guided token sampling, and an exploration-to-refinement schedule. A 7B student matches a 32B teacher in visual reasoning, with far better sample efficiency.

Catalog record

CREATORJongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65396
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron Abstract On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher–student log-likelihood ratio acts as a token reward. From this insight, we introduce Reopold ( Re laxed O n- Pol icy D istillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, Reopold temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, Reopold surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, Reopold outperforms recent RL approaches achieving 6.7 ∼ 12 × 6.7\sim 12\times greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ∼ 3.32 × \sim 3.32\times inference speedup. Figure 1 : Performance of Reopold . (a) Sample Efficiency : Reopold achieves a state-of-the-art trade-off between accuracy and sample efficiency on the AIME-25 benchmark. Detailed explanation can be found in Section 5.1 . (b) Test-time Scaling : On visual reasoning tasks, Reopold demonstrates superior test-time scaling capabilities compared to the vanilla RKL baseline. Notably, it allows smaller models to approach the performance of the 32B teacher. Detailed explanation can be found in Section 5.2 . \icml@noticeprintedtrue 1 Introduction Figure 2 : Illustration of Reopold . While standard on-policy distillation (a) often introduces instability and inefficiency by forcing the student to mimic the teacher excessively, our proposed Reopold (b) fosters a more stable and effective learning environment. By establishing a formal connection between distillation and RL via a stop-gradient operation (Section 3.1 ), Reopold utilizes teacher signals temperately (Section 4.1 ) and selectively (Section 4.2 ). As depicted, this approach filters out potentially harmful signals, preventing the student from deviating excessively from its original distribution. Large language models (LLMs) have achieved remarkable reasoning capabilities through reinforcement learning (RL) post-training and test-time scaling, exemplified by OpenAI’s o1/o3 ( 27 ) and DeepSeek-R1 ( 17 ) . However, replicating this success in small language models (SLMs) proves difficult. Due to limited representational capacity, SLMs struggle with direct reward optimization, rendering standard RL approaches ineffective ( 17 ; 11 ; 66 ) . This disparity necessitates alternative mechanisms specifically tailored for transferring reasoning abilities to capacity-constrained models. To address this, recent work ( 67 ; 45 ) adopts on-policy distillation, where the student learns from its own trajectories under the guidance of a high-capacity teacher, with superior reasoning capabilities. Unlike direct RL on sparse or high-variance rewards, this approach reduces optimization difficulty by constraining learning to regions aligned with the teacher’s behavior. Empirically, on-policy distillation has proven superior to RL algorithms and supervised fine-tuning (SFT) for reasoning tasks ( 39 ; 63 ) , offering a robust pathway for capability transfer. Despite its recent popularity, on-policy distillation lacks the methodological depth seen in modern RL. While RL for reasoning has rapidly evolved through specialized optimizers like GRPO ( 51 ) and mechanistic studies ( 50 ; 57 ) , distillation lacks comparable reasoning-centric advancements, despite efforts in standard instruction tuning ( 14 ; 29 ) . This imbalance suggests that on-policy distillation has yet to benefit from the advanced optimization techniques that have propelled RL forward, presenting a significant opportunity for refinement. Empirically, vanilla on-policy distillation suffers from instability ( 16 ; 14 ) . We observe that it frequently leads to negative transfer, where the student degrade compared to its initialization base model (Section 5.1 and Figure 7 ), and suffers from rapid entropy collapse which leads to premature convergence (Section 3.2 and Figure 5 ). Notably, similar limitations are echoed in recent literature ( 39 ) , which restricts teachers to matching sizes to ensure stability ( e.g., using an 8B teacher for an 8B student instead of 32B). Such constraints fundamentally limit the practical utility of current distillation. Contributions. We analyze on-policy distillation through the lens of RL to diagnose optimization instabilities. By interpreting the teacher–student log-likelihood ratio as a fixed reward, we cast distillation as policy optimization, leveraging modern RL insights to stabilize training. Our contributions are summarized as follows: • Diagnosing on-policy distillation: We demonstrate that stop-gradient renders the objective equivalent to standard policy gradient, acting as a control variate to mitigate variance and establish a robust baseline. Crucially, this allows us to diagnose bottlenecks, identifying heavy-tailed negative rewards and signal inefficiencies as the primary causes of instability. • Distillation-aware policy optimization: Addressing the optimization bottlenecks identified in our analysis, we propose Reopold , which filters out harmful distillation signals and softens aggressive updates by temperately and selectively applying learning signals from the teacher ( Figure 2 ). This method formulates a unified framework integrating reward clipping, token-level dynamic sampling, and multi-stage training. By explicitly regulating the learning signal, Reopold stabilizes the optimization where vanilla methods fail. • State-of-the-art efficiency and scalability: Empirically, Reopold achieves superior training sample efficiency and unlocks test-time scaling for SLMs ( Figure 1 ). It establishes state-of-the-art performance across mathematical, visual, and tool-use reasoning tasks, significantly outperforming standard baselines. Figure 3 : Comparison of training dynamics between vanilla RKL and RKL with stop-gradient. (a) Training loss dynamics exhibit similar trends, aligning with the theoretical equivalence in Remark 3.1 . (b) The gradient norm is markedly lower and more stable when stop-gradient is applied, which (c) leads to superior validation performance. This confirms that treating the log-likelihood ratio as a fixed reward signal is beneficial for optimization stability. 2 Background and Related Work 2.1 Reinforcement Learning for Reasoning Models Recent advancements have demonstrated that RL significantly enhances the reasoning capabilities of LLMs, such as DeepSeek-R1 ( 17 ) , OpenAI-o1/o3 ( 44 ) . In these frameworks, policy optimization typically relies on samples generated by a sampling policy ( e.g., a previous policy \pi_{\theta_{\text{old}}} ). Formally, given a query q drawn from a dataset \mathcal{Q} , a group of G responses \{o_{i}\}_{i=1}^{G} is sampled from \pi_{\theta_{\text{old}}} . The policy parameters \theta are then updated by maximizing the following objective: \begin{split}\mathcal{J}_{\text{RL}}(\theta)&=\mathbb{E}_{q\sim\mathcal{Q},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\\ &\left[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\rho_{i,t}(\theta)\hat{A}_{i,t}\right],\end{split} (1) where \rho_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t})} represents the importance ratio at token step t for response o_{i} , and \hat{A}_{i,t} denotes the estimated advantage. Note that we omit some clipping operations ( 49 ) here for brevity. These approaches generally employ scalable reward mechanisms based on final-answer correctness, exemplified by GRPO ( 51 ) and various parallelized RL methods ( 3 ; 32 ; 64 ) . Recent studies have proposed further refinements to improve stability and optimization dynamics. For instance, Dr. GRPO ( 37 ) eliminates variance normalization to mitigate bias, while DAPO ( 71 ) introduces a token-level loss and relaxes update constraints via expanded clipping thresholds. Other works explore modified clipping rules, enhanced normalization techniques, KL-regularization, and adaptive sampling strategies ( 10 ; 7 ; 57 ) . 2.2 On-Policy Distillation for Reasoning Models Distillation for LLMs is broadly categorized into off-policy and on-policy settings. In off-policy settings (e.g., SFT),

Read the complete text