Hybrid Policy Distillation for LLMs
Wenhong Zhu, Ruobing Xie, Rui Wang, Pengfei Liu · 2026
Read Hybrid Policy Distillation for LLMs by Wenhong Zhu, Ruobing Xie, Rui Wang, Pengfei Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20244.txt.
Introduction
Zhu and three colleagues reframe knowledge distillation for language models as a reweighted token-level likelihood goal, tying past methods together. Their Hybrid Policy Distillation blends forward and reverse KL, balancing broad mode cover with sharp mode seeking, and mixes off-policy data with light near on-policy sampling. Tests on long math reasoning plus short dialogue and code tasks show steadier optimization, lower compute cost, and stronger final scores across model families and sizes.
Catalog record
| CREATOR | Wenhong Zhu, Ruobing Xie, Rui Wang, Pengfei Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59556 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hybrid Policy Distillation for LLMs Wenhong Zhu, Ruobing Xie, Rui Wang, Pengfei Liu Abstract Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log-likelihood objective at the token level. We further propose Hybrid Policy Distillation (HPD) , which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, and combines off-policy data with lightweight, approximate on-policy sampling. We validate HPD on long-generation math reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. The code related to this work is available at https://github.com/zwhong714/Hybrid-Policy-Distillation . 1 Introduction Recent advancements in large language models (LLMs) have demonstrated remarkable performance across text generation tasks ( 43 ; 1 ) , largely driven by the massive scale of parameters and high-quality data ( 23 ) . However, such gains come at the cost of substantial computational and storage overhead ( 6 ) . Consequently, compressing LLMs by reducing the number of parameters while preserving their performance has become crucial for the practical deployment of these powerful models ( 13 ) . Knowledge distillation (KD) ( 20 ) has emerged as a principled approach for transferring knowledge from powerful teacher models to lightweight student models. In black-box KD, the student is limited to the teacher’s generation, and distillation is typically carried out via supervised fine-tuning (SFT) ( 16 ) . In contrast, white-box KD can exploit access to the teacher’s predictive distributions, training the student to match the teacher through distribution-level objectives such as Kullback–Leibler divergence (KLD) on logits ( 24 ) . Recent studies on LLM distillation further emphasize the importance of selecting appropriate divergence measures for effective distribution matching ( 8 ; 31 ; 25 ) . However, the choice of divergence alone is insufficient ( 51 ) . The behavior of divergence-based distillation for LLMs is governed by several tightly coupled design axes, including (i) the direction of divergence (forward vs. reverse). In practice, effective distillation requires balancing the complementary inductive biases of forward and reverse KL ( 5 ; 44 ) . Forward KL (FKLD) promotes mode coverage but often yields over-smoothed predictions ( 15 ) , whereas reverse KL (RKLD) emphasizes mode-seeking and distribution shaping, which can become unstable when the student–teacher gap is large ( 29 ) . (ii) The optimization strategy of the divergence (loss vs. reward), which leads to different optimization dynamics and gradient bias ( 3 ; 38 ) . (iii) The data regime (on-policy vs. off-policy). Off-policy distillation relies on an external data source that the student learns to imitate, but it often suffers from a train–inference behavior mismatch ( 2 ) . In contrast, on-policy distillation (OPD) samples rollouts from the student policy, which introduces a distribution shift on the teacher side for student-generated outputs and incurs substantial computational overhead ( 26 ) . These factors are often intertwined, jointly shaping the distillation behavior. Motivated by these intertwined design axes, we first formalize a reweighted log-likelihood objective to analyze SFT, FKLD, and RKLD. Under this view, FKLD and RKLD exploit the teacher’s full predictive distribution to provide dense supervision, at the cost of losing the computational efficiency of one-hot targets used in SFT. We then propose Hybrid Policy Distillation (HPD) , which simultaneously accounts for both divergence directions and on-/off-policy data regimes by treating divergences as token-level reward signals, while retaining the efficiency of one-hot supervision. Specifically, HPD computes token-level reweighting signals using a negative K_{1} estimator ( 37 ) applied to both offline tokens and student-sampled tokens. For offline tokens, a positive value indicates that the ground-truth token is under-estimated (i.e., the student assigns it too low a probability) and triggers forward-KL learning, whereas negative values are used to suppress over-estimation. For student-sampled tokens, over-estimation is also penalized with suppressed probability mass redistributed toward the offline expert tokens. This token-level hybridization combines the guidance of forward- and reverse-KL with the efficiency of one-hot supervision, while naturally handling both offline and lightweight on-policy sampling. We validate HPD on long-generation math reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. In summary, our contributions are as follows: • We present a unified reweighted log-likelihood view that facilitates understanding existing distillation methods and systematically enables the development of new approaches across divergence directions, optimization strategies, and data regimes. • We propose HPD, which hybridizes forward and reverse KL across off-policy data and lightweight on-policy sampling, retaining the efficiency of one-hot supervision. • Experiments on math reasoning, dialogue, and code tasks show that HPD improves optimization stability and efficiency, and consistently matches or outperforms standard SFT + OPD pipelines with less cost. Moreover, HPD provides a strong initialization for OPD, leading to further performance enhancement. 2 Problem Formulation Auto-regressive Language Models. We model next-token generation as a sequential decision process. Let \tau=(s_{1},a_{1}^{*},s_{2},a_{2}^{*},\dots,s_{T},a_{T}^{*}) denotes a trajectory from an offline dataset \mathcal{D} , where each state s_{t} represents the ground-truth prefix: s_{t}=\mathbf{a}_{<t}^{*}=(a_{1}^{*},\dots,a_{t-1}^{*}), and a_{t}^{*}\in\mathcal{V} is the expert token at step t drawn from a finite vocabulary \mathcal{V} . A pretrained LLM induces a policy q_{\theta}(a_{t}\mid s_{t}) parameterized by \theta , which factorizes the likelihood of the expert trajectory as: q_{\theta}(\tau)=\prod_{t=1}^{T}q_{\theta}(a_{t}^{*}\mid s_{t}). The model is pretrained via teacher forcing, by minimizing the expected negative log-likelihood (NLL) over \mathcal{D} as ( 33 ) : \mathcal{L}_{\text{pretrain}}(\theta)=-\mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{t=1}^{T}\log q_{\theta}(a_{t}^{*}\mid s_{t})\right]. (1) Limited Expressivity of the Student Model. The presence of multiple valid continuations, together with label smoothing, results in an inherently soft supervision signal ( 54 ) . Consequently, in highly complex output spaces, the teacher distribution or the offline data distribution often exhibits pronounced multimodality, exceeding the modes a capacity-limited student can represent ( 15 ) . Accordingly, distillation dilutes probability mass across too many modes, ultimately harming generation quality. 3 Preliminary We first review SFT as a special case of KD. While SFT provides sparse supervision on target tokens, KD leverages the teacher’s full predictive distribution to offer denser learning signals. However, exact divergence minimization is intractable in large-vocabulary settings, necessitating practical approximations such as Monte Carlo (MC) estimation. 3.1 KD via SFT SFT trains a model on an offline dataset \mathcal{D}=\{(s_{t},a_{t}^{*})\} by minimizing the cross-entropy loss. Specifically, the supervision signal is represented as a one-hot distribution \boldsymbol{y}_{t}\in\{0,1\}^{|\mathcal{V}|} , where y_{t,a_{t}^{*}}=1 and y_{t,a}=0 for all a\neq a_{t}^{*} . The resulting objective is \min_{\theta}\mathcal{L}^{\mathrm{SFT}}(\theta)=\min_{\theta}-{\mathbb{E}_{\boldsymbol{s}}\mathbb{E}_{\boldsymbol{a}\sim\mathcal{D}}}\left[\log q_{\theta}(\boldsymbol{a}|\boldsymbol{s})\right]. (2) While effective and stable, SFT provides supervision only at a single target action, yielding a sparse learning signal that ignores alternative plausible actions. 3.2 KD via KL Divergence By introducing the teacher distribution level signal, we can minimize a predictive divergence \mathbb{D} between the teacher distribution and the parameterized student distribution. The common use is FKLD and RKLD. FKLD. Forward KL measures how well the student distribution q_{\theta}(\boldsymbol{a}\mid\boldsymbol{s}) covers the modes of the teacher distribution p(\boldsymbol{a}\mid\boldsymbol{s}) , defined as follows ( 24 ) : \min_{\theta}\mathbb{D}_{\mathrm{KL}}(p\|q_{\theta})=\min_{\theta}\mathbb{E}_{\boldsymbol{s}}\mathbb{E}_{\boldsymbol{a}\sim p(\cdot|\boldsymbol{s})}\left[\log\frac{p(\boldsymbol{a}|\boldsymbol{s})}{{\color[rgb]{0,0,1}q_{\theta}(\boldsymbol{a}|\boldsymbol{s})}}\right]. (3) Recent studies ( 2