OISD: On-Policy Internal Self-Distillation of Language Models

Xinyu Liu, Darryl Cherian Jacob, Yang Zhou, Jindong Wang, Pan He · 2026

Read OISD: On-Policy Internal Self-Distillation of Language Models by Xinyu Liu, Darryl Cherian Jacob, Yang Zhou, Jindong Wang, Pan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29089.txt.

Introduction

Xinyu Liu and team present on-policy inner self-distillation for language models, called OISD. During rollout and GRPO tuning, the final layer acts as policy and solo teacher for selected middle layers. Two tools align middle layers with the final layer: logit alignment for thinking habits, and attention match for attention patterns, both without special information. Advantage-weighted Jensen-Shannon training trains middle layers while keeping policy match under one acting policy. Tests show great gains over strong thinking rivals across four math thinking tasks.

Catalog record

CREATORXinyu Liu, Darryl Cherian Jacob, Yang Zhou, Jindong Wang, Pan He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47681
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29089

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OISD: On-Policy Internal Self-Distillation of Language Models Xinyu Liu, Darryl Cherian Jacob, Yang Zhou, Jindong Wang, Pan He Abstract Recent reinforcement learning (RL) post-training approaches primarily optimize the final output policy using sparse outcome-level rewards, while largely overlooking predictive signals encoded in intermediate representations. In this paper, we introduce a new paradigm called on-policy internal self-distillation and propose the OISD framework, which improves reasoning by transferring on-policy predictive signals from the final layer to intermediate representations. During rollout and Group Relative Policy Optimization (GRPO) optimization, the final layer acts as both the policy and a detached internal teacher for selected intermediate layers, which are guided to align with it through two complementary mechanisms: logit alignment, which transfers high-level reasoning behaviors ( how to think ), and attention alignment, which enforces consistent attention patterns ( where to look ) from the final layer to the selected intermediate layer, both without requiring external privileged information. Our OISD , together with GRPO, employs signed advantage-weighted Jensen–Shannon alignment to distill informative intermediate representations while preserving policy consistency under a unified acting policy. Experimental results demonstrate the effectiveness of OISD , with substantial and consistent improvements over strong reasoning RL baselines across four mathematical reasoning tasks. The code will be released at https://github.com/THE-MALT-LAB/OISD 1 Introduction Figure 1: Comparison of four distillation paradigms: (a) Off-Policy Distillation, (b) On-Policy Distillation, (c) On-Policy Self-Distillation, and (d) On-Policy Internal Self-Distillation (Ours). The comparison highlights differences in teacher–student roles, rollout generation, and the location where distillation is applied. Large language models (LLMs) have achieved remarkable advances in mathematical reasoning, coding, and complex instruction-following ( 35 ; 31 ) . A key driver of this progress is reasoning-oriented post-training, in which models are further optimized after pretraining and supervised fine-tuning to generate more accurate, coherent, and reliable reasoning trajectories ( 26 ; 9 ) . Recent reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO)-style training, has significantly improved reasoning-oriented post-training without requiring dense human supervision ( 26 ; 9 ; 40 ; 37 ) . However, these approaches remain heavily final-output-centric, optimizing only the final acting policy with sparse outcome rewards while largely overlooking intermediate predictions, reasoning processes, and internal representations. Knowledge distillation ( 12 ; 4 ) provides dense token-level supervision along the reasoning trajectory, complementing sparse outcome rewards. In classical off-policy distillation, the student learns from fixed teacher-generated traces, creating a distribution mismatch because the training trajectories come from the teacher rather than the student’s own generations ( 1 ; 32 ) . On-policy distillation reduces this gap by supervising student-generated rollouts directly ( 1 ) , but most methods still rely on external teacher models. Recent on-policy self-distillation methods remove the external teacher by augmenting the same model with privileged traces, feedback, or auxiliary prompts ( 39 ) . However, supervision still depends on externally injected privileged information rather than the model’s own internal computations. Existing off-policy and on-policy distillation methods primarily focus on improving model accuracy and reasoning capability, with limited exploration of leveraging the model’s own internal mechanisms for distillation. However, prior studies ( 34 ; 42 ) show that reasoning representations evolve systematically across layers: earlier layers preserve broader candidate predictions, intermediate layers integrate contextual and task-relevant information, and deeper layers progressively converge toward the final next-token decision. This hierarchical organization suggests that different layers serve distinct functional roles in the reasoning process. Meanwhile, the logit lens framework ( 22 ) provides initial insights by projecting intermediate representations into token space using the unembedding matrix. This reveals that the residual stream in Transformer architectures contains rich predictive information that evolves progressively across layers and modules ( 7 ; 10 ) . All these insights motivate leveraging the model’s own internal computations as an intrinsic source of supervision for distillation. In this paper, we propose OISD , an on-policy internal self-distillation framework for LLM reasoning. During training, OISD keeps the final layer as the sole acting policy for rollout and GRPO optimization, while using it as a detached internal teacher. Following the logit lens framework ( 3 ) , a selected intermediate layer is projected into the token space to form an auxiliary internal student on the same on-policy rollout. The student learns from two complementary signals: logit alignment, which transfers high-level reasoning behaviors ( how to think ), and attention alignment, which enforces consistent attention patterns ( where to look ) from the final layer to the selected intermediate layer, both without requiring external privileged information. Such a formulation encourages the model to learn stronger intermediate representations. During inference, the teacher model is used as the final model. OISD introduces dense internal supervision without requiring an external teacher, privileged prompts, or multiple acting policies, motivating the term on-policy internal self-distillation . A comparison between our OISD and other methods is shown in Figure 1 . 1.1 Contributions Our contributions are summarized as follows: • We introduce a new paradigm called on-policy internal self-distillation for reasoning RL that leverages the model’s own internal computations as supervision while preserving a single acting policy for rollout and optimization. • We propose OISD , an on-policy internal self-distillation framework that uses the final layer as a detached internal teacher and an intermediate layer as an auxiliary internal student and transfers on-policy predictive signals from the final layer to intermediate representations. • We develop two complementary internal alignment objectives: logit alignment for how to think by transferring predictive beliefs, and attention alignment for where to look by transferring evidence-routing behavior. • We demonstrate substantial and consistent improvements over strong baselines across four mathematical reasoning benchmarks. We hope this research initiates the investigation of on-policy internal self-distillation and encourages future research in this direction. 2 Related Work Knowledge Distillation for Reasoning. Recent Reinforcement Learning from Human Feedback (RLHF) methods based on PPO provide a standard framework for preference optimization ( 25 ; 23 ) . Later, reasoning-oriented RL methods, such as GRPO, DeepSeek-R1, DAPO, GSPO, Qwen3, and RAIF, leverage verifiable rewards and group-based policy optimization to improve long context reasoning ( 26 ; 9 ; 40 ; 33 ; 24 ; 37 ) . However, supervision remains largely sparse, outcome-level, and focused on the final acting policy. To provide denser reasoning supervision, prior work ( 12 ; 4 ) often distills teacher distributions or reasoning traces into the student. Yet off-policy distillation introduces a training–inference mismatch, since the student is trained on teacher-generated trajectories but must reason from its own internal states during inference. On-policy distillation reduces this mismatch by generating rollouts from the student and using the teacher’s per-token log-probabilities as dense supervision on states actually visited by the student ( 1 ; 32 ) . More recent self-distillation methods reduce reliance on external teachers by enabling the model to supervise itself using privileged information or richer feedback ( 1 ; 13 ; 28 ) . For example, OPSD 39 leverages privileged solutions, SDPO 13 exploits environment feedback, OPCD ( 36 ) distills context-conditioned behavior into model weights, and CODI 27 compresses explicit reasoning into continuous representations. Our method places the teacher entirely within the model itself: the detached final layer supervises an intermediate layer on the same on-policy rollout while remaining the sole acting policy.

Read the complete text