Distribution Corrected Offline Data Distillation for Large Language Models
Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu · 2026
Read Distribution Corrected Offline Data Distillation for Large Language Models by Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14071.txt.
Introduction
Zhang and three co-authors tackle distributional drift in offline distillation of reasoning traces for language models. Teacher traces give high-quality, sample-efficient supervision, but students train on teacher-written prefixes while inferring on self-written ones, so errors compound over long reasoning chains. On-policy or self-distillation methods match inference conditions better, yet they demand costly online sampling and yield weak traces early in training. The paper proposes a principled offline framework that corrects the distribution mismatch without going online.
Catalog record
| CREATOR | Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82129 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14071
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distribution Corrected Offline Data Distillation for Large Language Models Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu Abstract Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings. Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift: during training, the student model conditions on teacher-generated prefixes, whereas during inference the student autoregresses on self-generated prefixes, leading to compounding errors over long reasoning trajectories. Meanwhile, on-policy or self-distillation methods better match the student’s inference-time distribution, but require costly online sampling and often produce low-quality traces in early training. We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher–student distribution drift. It adaptively emphasizes teacher supervision that is better aligned with the student’s on-policy distribution. Evaluations on mathematical reasoning benchmarks of GSM8K, MATH, MATH500, and harder held-out competition-style tasks, including AMC, AIME, and OlympiadBench, show that our method improves reasoning accuracy over prior offline distillation algorithms and yields more stable reasoning traces while preserving instruction-following capabilities. Our work shows that lightweight, distribution-correction-aware training can substantially strengthen offline reasoning distillation without online rollouts. 1 1 1 Code is available at https://anonymous.4open.science/r/Discord-E6BC . 1 Introduction Reasoning ability has emerged as a cornerstone of modern large language models (LLMs), largely driven by scaled model architecture and advances in reinforcement learning (RL)-based post-training optimization. This capability is particularly important in scientific and engineering domains such as mathematics, robotics, and physics, where solving problems requires structured, multi-step reasoning instead of short-horizon token prediction. Despite these advances, high-quality reasoning behavior is still primarily observed in large or proprietary models. Enabling smaller models to acquire similar reasoning ability is critical for practical deployment, especially in resource-constrained environments where efficient and reliable intelligence is required. The ability to transfer reasoning competence across models without significant performance degradation is crucial. A common approach to reasoning distillation is to train a smaller student model on reasoning traces generated by a stronger teacher model. In this paradigm, referred to as offline learning or behavior cloning , training data are collected from an external model. In contrast, on-policy methods generate data from the target model itself and update the model using policy-gradient RL techniques such as GRPO [ 30 ] or token-level extensions [ 21 , 42 , 46 , 43 ] . Offline distillation offers several advantages. It is typically more sample-efficient, and teacher-generated trajectories provide higher-quality supervision, especially during early stages of training. However, it also faces the inherent limitation of distribution drift : the distribution of teacher-generated trajectories differs from the distribution induced by the student model during inference. As the student generates intermediate tokens that deviate from teacher trajectories, it enters support regions that are underrepresented in the training data. This distribution shift leads to compounding errors and unreliable behavior. Existing approaches alleviate this issue through refined distillation objectives, such as bidirectional divergence minimization [ 17 ] , but these methods do not directly resolve the distribution mismatch, where the optimization is still focused on a teacher-provided distribution. We propose a principled method for LLM reasoning distillation that relies solely on offline, teacher-provided data while reducing the distribution gap to on-policy training. We first design a general reasoning-eliciting instruction that contextualizes a large teacher model to produce high-quality reasoning traces. These traces serve as supervision for training a smaller student model. We then introduce a distribution-corrected objective that incorporates an advantage-based weighting mechanism, enabling the student to better align with trajectories it is likely to encounter during inference. Our formulation provides an effective approach for reducing the teacher–student distributional drift and is compatible with most offline distillation frameworks. We focus on mathematical reasoning as a representative domain. Empirical results across multiple models and math benchmarks demonstrate that our method consistently improves the reasoning performance of student models in both in-domain and out-of-domain settings, outperforming existing offline distillation baselines and narrowing the gap to that of online sampling methods without requiring online interaction. 2 Preliminary On-policy Learning. On-policy methods, especially on-policy RL, have been widely adopted for LLM post-training in domains such as math reasoning and safety alignment. The core idea behind this approach is to fine-tune a pretrained model \pi_{\theta} to maximize accumulated rewards within a domain \mathcal{Q} over self-generated trajectories: J_{\text{RL}}(\theta)=\mathbb{E}_{x\sim\mathcal{Q},\;y\sim\pi_{\theta}(\cdot\mid x)}\left[\mathds{R}(y,x)\right], (1) where \mathds{R}(x,y) is a reward function that scores a response y ( e.g. , a reasoning trajectory). Following the policy gradient principle [ 32 ] , we derive the following gradient update for Eq. ( 1 ): \nabla_{\theta}J_{\text{RL}}(\theta)=\mathbb{E}_{x\sim\mathcal{Q},\;\underbrace{y\sim\pi_{\theta}(\cdot\mid x)}_{\text{On-policy sampling}}}\left[\mathds{R}(x,y)\cdot\sum_{t=0}^{|y|}\log\pi_{\theta}(y_{t}\mid x,\,y_{:<t})\right], (2) Prior work has improved training stability by replacing the reward function \mathds{R}(x,y) with a normalized Advantage function, tokenizing the optimization formulation, and introducing clipping and model update constraints [ 30 , 7 , 21 , 42 , 46 , 43 ] . A key advantage of this strategy over supervised fine-tuning (SFT) lies in its on-policy learning: the data used to optimize the target model is sampled from itself instead of from another behavior policy. It mitigates distribution drift, where the model encounters underrepresented regions at inference time, and often benefits out-of-domain generalization. However, online methods are costly in terms of sample collection and labeling, requiring a notably higher number of training samples than supervised fine-tuning, and each trajectory requires labeled rewards. Additionally, for smaller LLMs, the sample quality is also limited by the model’s own capability, which leads to rare high-return trajectories and slow asymptotic convergence. Offline Knowledge Distillation. A parallel path towards improving pretrained LLM performance is knowledge distillation from another stronger model, such as an advanced teacher LLM \pi_{T} . One straightforward method is to directly perform SFT on teacher-generated reasoning traces, which trains the target (student) model \pi_{\theta} to imitate teacher behavior by maximizing the log-likelihood of teacher-generated responses: J_{\text{SFT}}(\theta)=\mathbb{E}_{x\sim\mathcal{Q},\;\underbrace{y\sim\pi_{T}(\cdot\mid x)}_{\text{off-line sampling}}}\left[\sum_{t=0}^{|y|}\log\pi_{\theta}(y_{t}\mid x,\,y_{:<t})\right]. (3) Here the expectation is taken under the teacher distribution \pi_{T} , with y drawn from an offline dataset provided by the teacher model. This behavior-cloning style method is training- and sampling-efficient but is also prone to performance drop induced by distribution drift, since this offline objective does not directly optimize the target ( \pi_{\theta} ) model’s performance when y is sampled from \pi_{\theta} during inference. Subsequent methods supervise the student model with more refined teacher signals, often by minimizing a distributional divergence between the teacher and student models, such as the KL-divergence: \displaystyle J_{\text{KD}}(\theta)=\mathbb{E}_{x\sim\mathcal{Q},\;y\sim\pi_{T}(\cdot\mid x)}\left[\mathcal{D}_{\text{KL}}(\pi_{T}(y|x)|\pi_{\theta}(y|x))\right]. (4) Works along this line include optimizing on a reverse KL divergence \mathcal{D}_{\text{KL}}(\pi_{\theta}\|\pi_{T}) [ 4 ] or a mixture of both directions [ 17 ] . While these algorithms show empirical performance improvement over direct SFT, they still inherit a fundamental limitation of offline data: the training objective provides a biased signal for the student’s on-policy behavior. At inference time, the student model may generate prefixes that are underrepresented in the offline teacher data, leading to unreliable behavior [ 1 ] . Figure 1: Overview of DisCorD : Standard offline distillation imitates fixed teacher traces under teacher-generated prefixes, but during inference the student conditions on its own prefixes, causing distribution drift and compounding errors (left). DisCorD reduces this mismatch by reweighting offline teacher-token supervision according to