Online Knowledge Distillation with Reward Guidance
Chen Jia · 2025
Read Online Knowledge Distillation with Reward Guidance by Chen Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18952.txt.
Introduction
Chen Jia teaches small language models from big ones with a referee's help in a 2025 paper. Distillation copies a teacher model's skill into a student, but plain copying ignores which answers users actually prefer. Preference training fixes taste; merging the two was unsettled.
The framework sets a contest: the student policy tries to close the gap to the teacher while a reward model, kept honest within a confidence band, scores the attempts. Preference data can be gathered offline or live during training. A reformulation ties the referee to action values, and a white-box version peeks at the teacher's raw odds. Theory and tests both favor the guided student.
Catalog record
| CREATOR | Chen Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Online Knowledge Distillation with Reward Guidance Chen Jia Abstract This work studies knowledge distillation (KD) for large language models (LLMs) through preference optimization. We propose a reward-guided imitation learning framework for sequential KD, formulating a min-max optimization problem between the policy and reward model (RM) to minimize the performance gap between the student and teacher policies. Specifically, the reward optimization is constrained to achieve near-optimality within a confidence set for preference alignment. For preference data construction, we explore both offline and online preference-based KD. Additionally, we reformulate the RM using the Q Q -value function and extend the framework to white-box KD, where the teacher policy’s predicted probabilities are accessible. Theoretical analysis and empirical results demonstrate the effectiveness of the proposed framework. 1 Introduction Large language models (LLMs) such as GPT-4 [ 2 ] and PaLM [ 4 ] have demonstrated impressive capabilities across a wide range of natural language processing (NLP) tasks. Their strong generalization performance is largely attributed to large-scale pretraining and increased model capacity. However, these benefits come at the cost of substantial computational and memory demands, which pose significant challenges for real-world deployment, particularly in resource-constrained settings. To address these challenges, knowledge distillation (KD) [ 23 ] has become a widely adopted technique to compress large teacher models into smaller, more efficient student models, while aiming to preserve much of the teacher’s performance. KD methods can be broadly categorized into black-box and white-box approaches, depending on the accessibility of the teacher model’s internal outputs. In black-box KD, the student learns by mimicking the teacher’s outputs using maximum likelihood estimation (MLE) [ 28 ] . In white-box KD, the student has access to the teacher’s output distributions, allowing for more informative training through distribution matching [ 23 , 32 , 52 , 20 , 3 ] . Despite their success, existing KD methods face two major challenges. First, the large capacity gap between teacher and student models makes it difficult for the student to accurately replicate the teacher’s outputs, particularly in white-box KD. Second, the teacher’s outputs may themselves be suboptimal for specific downstream tasks, making it undesirable to blindly mimic them. To overcome these limitations, we propose a novel preference-based knowledge distillation (PbKD) framework as shown in Figure 1 . Instead of directly cloning the teacher’s behavior, PbKD formulates KD as a reward-guided imitation learning problem. Our method optimizes the performance gap between the student and teacher policies through a min-max game between the student and a reward model (RM). The RM is constrained within a confidence set constructed from preference data and is optimized to maximize the distinguishability between the student and teacher, guiding the student to improve accordingly. We further develop an online PbKD variant that iteratively augments the preference dataset with new samples collected from the evolving student policy, enabling continual refinement of the RM and more accurate alignment. In the white-box setting, we extend PbKD by reformulating the RM using Q -functions to more effectively leverage the teacher’s output probabilities. We provide theoretical guarantees for both the offline and online settings. For offline PbKD, we derive a suboptimality bound with a convergence rate of \mathcal{O}(\sqrt{\log(N/\delta)/N}) , and for online PbKD, we prove a regret bound of order \mathcal{O}(\sqrt{T\log T\log(T/\delta)}) , where N is the number of preference samples, d is the feature dimension, T is the number of iterations, and \delta is the confidence level. Extensive empirical evaluations across five black-box and five white-box KD benchmarks confirm the effectiveness of our approach, outperforming prior methods. Figure 1: We formulate reward-guided imitation learning as the optimization of the performance gap between the teacher policy \pi_{E} and the student policy \hat{\pi} , defined as \hat{\pi}:=\mathop{\arg\min}_{\pi}\max_{r}J(\pi_{E},r)-J(\pi,r) (Eq. ( 3 )). In the offline PbKD setting, a pre-collected preference dataset is used to construct a confidence set for constraining the reward model with MLE, i.e., r\in\mathcal{R}(\mathcal{D}^{\rm pref}_{\rm off}) (Algorithm 1 ). In the online PbKD setting, new preference data are iteratively collected from the current student policy and incorporated into the confidence set, resulting in time-dependent constraints: r\in\mathcal{R}(\mathcal{D}^{\rm pref}_{t}) at each iteration t\in\{1,2,\ldots,T\} (Algorithm 2 ). 2 Related Work Black-Box Knowledge Distillation. Black-box knowledge distillation (KD) has emerged as a practical solution for compressing large language models (LLMs) like GPT-4 [ 7 ] , Claude 3 [ 5 ] , and Gemini (Google, 2023) into smaller, more efficient student models. Black-box KD leverages only the output responses of proprietary APIs. Recent efforts such as Alpaca [ 45 ] , Vicuna [ 12 ] , and Orca [ 35 ] have explored using these responses as supervision to transfer capabilities like reasoning, tool use, and code generation. Techniques include prompting LLMs to generate Chain-of-Thought (CoT) explanations [ 25 , 49 ] rationale-based reasoning [ 34 ] , or code-language pairs [ 6 ] , which are then used to fine-tune smaller models. These approaches demonstrate the potential of black-box LLMs as data generators to train open-source alternatives without requiring access to their internal weights or training data. White-Box Knowledge Distillation. White-box knowledge distillation (KD) builds upon the classical KD framework introduced by Hinton et al. [ 23 ] , where the student model learns from the soft predictions of a larger teacher model. This paradigm has been widely applied to compress large pre-trained language models (PLMs), with early work focusing on logits and intermediate features [ 27 , 42 , 48 ] . Subsequent methods extended KD to sequence generation tasks, including machine translation and summarization, by aligning entire output distributions [ 11 , 47 ] . Recent efforts target large language models (LLMs) specifically, exploring divergence-based objectives such as KL, reverse KL (RKL), and Jensen-Shannon (JS) divergence for auto-regressive generation [ 52 , 3 , 20 ] . For instance, Gu et al. [ 20 ] introduced a policy gradient method to reduce RKL variance, while Ko et al. [ 29 ] proposed an efficient skew-divergence-based technique. Additionally, moment-matching and step-wise supervision have gained attention for better aligning generation dynamics [ 26 ] . These works reveal the limitations of traditional divergence objectives and highlight the need for more adaptive and data-efficient strategies in white-box KD for LLMs. Preference-based Knowledge Distillation. While knowledge distillation (KD) has been extensively studied for large language models (LLMs), relatively few efforts have explored the use of preference optimization to guide the distillation process, aiming to align student models with human preferences. Recent works on preference-based knowledge distillation (PbKD) include [ 30 , 58 , 9 ] , where a preference loss is incorporated—typically using pairwise comparisons between teacher and student outputs—alongside the conventional maximum likelihood estimation (MLE) loss used in behavior cloning. Notably, Li et al. [ 30 ] and Chen et al. [ 9 ] focus on black-box settings, whereas Zhang et al. [ 58 ] propose a method that falls under white-box KD. In contrast to these approaches, we propose a more general PbKD framework grounded in imitation learning, applicable to both black-box and white-box scenarios. Our improvements stem from a more robust min-max optimization framework with an online, iterative training scheme that enhances preference alignment and training stability. 3 Problem Formulation In this section, we formulate sequential knowledge distillation as an imitation learning problem for language generation.