Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

Siyan Zhao †, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen ∗,‡, Aditya Grover ∗ · 2026

Read Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models by Siyan Zhao †, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen ∗,‡, Aditya Grover ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18734.txt.

Introduction

Siyan Zhao and six colleagues let a language model teach it-self to think, with its own tries as the class. In the plain school path, a big pro model writes the steps and a small model learns to copy them. That path has a flaw: the small model trains on steps it would not take, so its live acts drift from its training. A fix lets the small model write its own tries while the pro grades each step.

Their plan goes one step more: one model plays both roles. It writes a try, then grades its own steps with a pro view of the same weights. No next model is needed at all. A self-taught loop could cut the cost of thought training a lot. It could also keep the training in step with how the model truly acts.

Catalog record

CREATORSiyan Zhao †, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen ∗,‡, Aditya Grover ∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models Siyan Zhao †, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen ∗,‡, Aditya Grover ∗ Abstract Knowledge distillation improves large language model (LLM) reasoning by compressing the knowledge of a teacher LLM to train smaller LLMs. On-policy distillation advances this approach by having the student sample its own trajectories while a teacher LLM provides dense token-level supervision, addressing the distribution mismatch between training and inference in off-policy distillation methods. However, on-policy distillation typically requires a separate, often larger, teacher LLM and does not explicitly leverage ground-truth solutions available in reasoning datasets. Inspired by the intuition that a sufficiently capable LLM can rationalize external privileged reasoning traces and teach its weaker self (i.e., the version without access to privileged information), we introduce On-Policy Self-Distillation (OPSD), a framework where a single model acts as both teacher and student by conditioning on different contexts. The teacher policy conditions on privileged information (e.g., verified reasoning traces) while the student policy sees only the question; training minimizes the per-token divergence between these distributions over the student’s own rollouts. We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks, achieving 4-8× token efficiency compared to reinforcement learning methods such as GRPO and superior performance over off-policy distillation methods. † † affiliationnotice: Equal contribution, \siyanmeta , \workatmeta 1 Introduction Figure 1 : Overview of On-Policy Self-Distillation (OPSD): Given a reasoning dataset \mathcal{S}=\{(x_{i},y_{i}^{\star})\}_{i=1}^{N} , we instantiate two policies from the same LLM: a student policy p_{S}(\cdot\mid x) and a teacher policy p_{T}(\cdot\mid x,y^{\star}) . The student generates an on-policy response \hat{y}\sim p_{S}(\cdot\mid x) . Both policies then evaluate this trajectory to produce next-token distributions p_{S}(\cdot\mid x,\hat{y}_{<n}) and p_{T}(\cdot\mid x,y^{\star},\hat{y}_{<n}) at each step n . The learning objective minimizes the per-token divergence D(p_{T}\|p_{S}) along the student’s rollout. Crucially, gradients backpropagate only through the student’s logits, allowing the model to self-distil. Recent advances in large language models (LLMs) have demonstrated impressive capabilities in reasoning and instruction following. Achieving these capabilities during post-training typically relies on reinforcement learning methods such as Reinforcement Learning with Verifiable Rewards (RLVR) (e.g., GRPO ( 28 ; 10 ; 32 ; 24 ; 42 ) ), supervised fine-tuning (SFT) on high-quality reasoning datasets ( 8 ; 32 ; 36 ) , or knowledge distillation, where recent work has shown that distillation from advanced teacher models can outperform RL in both performance and training efficiency ( 39 ; 36 ; 19 ) . Despite their respective successes, each approach has inherent limitations. RLVR suffers from inefficiencies including: (1) sampling a group of responses per prompt is computationally expensive and can introduce high variance in estimating the true value function; moreover, when all samples are either correct or incorrect, the gradient signal vanishes ( 42 ; 47 ) ; and (2) the reward signal is sparse and uniformly applied across all tokens in the generated output, neglecting fine-grained token-level feedback. Supervised fine-tuning suffers from exposure bias and weaker generalization ( 1 ; 6 ) . Traditional knowledge distillation provides dense token-level supervision from a teacher model but relies on off-policy data ( 11 ) . Recent advances in on-policy distillation—where a student model samples its own trajectories while a teacher policy provides dense token-level supervision—have demonstrated superior sample efficiency by combining the distributional realism of on-policy training with dense feedback ( 1 ; 19 ) . While on-policy distillation has shown strong performance, it relies on a distinct teacher model to supervise the student. Given that modern LLMs already exhibit strong reasoning capabilities, we ask this research question: can a model effectively serve as its own teacher through self-distillation? Our approach is inspired by human learning: after solving a problem incorrectly, a student can examine the correct solution, rationalize its steps, and identify where their reasoning failed. Prior work has shown that for LLMs, evaluation is often easier than generation ( 31 ; 21 ) . We hypothesize that rationalization —explaining a given correct answer—is similarly easier than generation. Motivated by this, we instantiate both the teacher and student policies from a single LLM. The teacher policy is provided with privileged information y^{\star} , such as the ground-truth answer or a reference chain-of-thought, while the student policy conditions only on the problem x . Concretely, the teacher policy p_{T}(\cdot\mid x,y^{\star}) conditions on both the problem and the privileged answer, whereas the student policy p_{S}(\cdot\mid x) observes only the problem. We preserve the on-policy training paradigm by sampling trajectories \hat{y} exclusively from the student policy, which then receives dense, token-level supervision from the privileged teacher policy. We therefore propose On-Policy Self-Distillation (OPSD) , a framework in which a single model plays both teacher and student roles. The student samples its own trajectories \hat{y}\sim p_{S}(\cdot\mid x) ; we then compute the per-token divergence between the student and teacher distributions and minimize it over the student’s own rollouts. This formulation (i) uses on-policy supervision (the student’s own trajectories), (ii) provides dense per-token feedback, (iii) exploits ground-truth solutions y^{\star} , and (iv) requires no separate teacher model. The learning process is captured by the loss \displaystyle\mathcal{L}_{\mathrm{OPSD}} \displaystyle(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\;\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\sum_{n=1}^{|\hat{y}|} \displaystyle\quad D\!\Bigl(p_{T}\!\left(\cdot\mid x,y^{\star},\hat{y}_{<n}\right)\;\Big\|\;p_{S}\!\left(\cdot\mid x,\hat{y}_{<n}\right)\Bigr). (1) In summary, our contributions are as follows: • We introduce On-Policy Self-Distillation, a novel framework that enables a single model to act as both teacher and student, leveraging ground-truth answers to provide dense token-level supervision on student rollouts. • We evaluate OPSD on four competition-level mathematical reasoning tasks, demonstrating that it outperforms both RLVR (e.g., GRPO) and supervised fine-tuning baselines. • We show that OPSD achieves better performance with nearly 8\times improved token efficiency and lower computational cost than GRPO. • We analyze the impact of model scale, finding that moderate model capacity is necessary for successful self-distillation. We further compare different divergence objectives and analyze the effect of student generation length. SFT/Off-Policy GRPO On-Policy On-Policy Distillation Distillation Self-Distillation (Ours) On-Policy Data ✗ ✓ ✓ ✓ Dense Learning Signal ✓ ✗ ✓ ✓ Low Sampling Cost ✓ ✗ ✓ ✓ No External Teacher ✓ ✓ ✗ ✓ Table 1 : Comparison of training methods for reasoning tasks. On-Policy Self-Distillation (OPSD) combines the advantages of on-policy training with dense feedback without requiring an external teacher model. 2 Background 2.1 Knowledge Distillation for Autoregressive Large Language Models Knowledge distillation transfers knowledge from a larger teacher model to a smaller student model by training the student to mimic the teacher’s behavior ( 11 ; 14 ; 26 ) . The core insight is that the teacher’s soft probability distribution over classes contains richer information than hard labels alone, as it reveals the teacher’s learned similarities between classes. For auto-regressive language models, given a dataset \mathcal{S}=\{(x,y^{\star})\} where x denotes an input and y^{\star} is the corresponding reference output, both teacher p_{T} and student p_{S} define token-level distributions over vocabulary \mathcal{V} . Traditional supervised distillation minimizes a divergence D between teacher and student distributions averaged over a fixed dataset: \mathcal{L}_{\text{Supervised Distillation}}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{S}}[D(p_{T}\|p_{S})(y|x)], (2) where D(p_{T}\|p_{S})(y|x)=\frac{1}{|y|}\sum_{n=1}^{|y|}D(p_{T}(\cdot|y_{<n},x)\|p_{S}(\cdot|y_{<n},x)) measures per-token discrepancy. However, this off-policy approach suffers from distribution mismatch: the student encounters different partial sequences y_{<n} during auto-regressive generation at inference than those seen during training on the fixed dataset, leading to compounding errors. On-policy distillation ( 1 ; 19 ; 37 ) addresses this

Read the complete text