Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen · 2026

Read Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation by Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00223.txt.

Introduction

Luong and two colleagues diagnose a flaw in reverse KL distillation for large language models. Non-target gradients keep pushing the target logit upward even after the student matches the teacher, breeding overconfidence and shrinking output diversity. Reverse KL also supervises non-target classes weakly, harming tail alignment. Their Diversity-aware RKL cancels the harmful gradient push and firms up non-target supervision, keeping reverse KL's optimization advantages. Across datasets and model families, it beats forward KL, reverse KL, and other objectives on performance and the trade-off.

Catalog record

CREATORHoang-Chau Luong, Dat Ba Tran, Lingwei Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59158
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation Hoang-Chau Luong, Dat Ba Tran, Lingwei Chen Abstract Reverse Kullback–Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher–student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL introduces a structural limitation that drives the student toward overconfident predictions. We first provide an analysis of RKL by decomposing its gradients into target and non-target components, and show that non-target gradients consistently push the target logit upward even when the student already matches the teacher, thereby reducing output diversity. In addition, RKL provides weak supervision over non-target classes, leading to poor tail alignment. To address these issues, we propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across datasets and model families demonstrate that DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity–diversity trade-off. 1 Introduction Knowledge distillation (KD) ( 9 ) aims to transfer knowledge from large language models (LLMs) to smaller, more efficient students ( 25 ) , typically by minimizing a Kullback–Leibler (KL) divergence between their output distributions. While forward KL (FKL) has been the standard objective in early distillation work ( 9 ; 4 ; 18 ; 11 ; 20 ) , recent LLM distillation methods increasingly favor reverse KL (RKL) due to its superior empirical performance ( 7 ; 8 ; 1 ) . This advantage is commonly attributed to the mode-seeking behavior of RKL, which concentrates probability mass on dominant teacher modes, whereas FKL is more mass-covering and encourages matching the full distribution ( 2 ) . However, prior work ( 26 ) suggests that FKL and RKL converge to the same optimum under sufficient optimization, implying that their differences mainly manifest during early training. In this work, we argue that the advantage of RKL arises from its alignment with the structure of LLMs, rather than being limited to the early stages of training. The theoretical equivalence between FKL and RKL relies on assumptions (e.g., sufficient student capacity, exact optimization to the global optimum) that rarely hold in modern LLM distillation, particularly under large vocabularies and significant teacher–student capacity mismatch. In such settings, FKL enforces probability matching across the entire vocabulary, requiring the student to approximate the teacher’s long-tail distribution, which is inherently challenging for capacity-limited models. In contrast, RKL focuses learning on high-probability tokens while suppressing low-probability mass, effectively reducing the complexity of the alignment problem. This leads to the better performance of RKL observed in practice ( 7 ; 12 ) . Despite these advantages, RKL introduces fundamental limitations. First, it creates a shortcut that rapidly minimizes the non-target loss by increasing the confidence of the target class, without properly aligning the non-target distributions of the teacher and student. As a result, knowledge transfer is incomplete, since the RKL objective can remain low despite poor non-target alignment. Second, by concentrating probability mass on dominant tokens, it systematically reduces the entropy of the student distribution, resulting in overconfident predictions and diminished output diversity. Empirically, students trained with RKL often exhibit higher confidence than their teachers, creating a clear quality–diversity imbalance. To better understand these behaviors, we theoretically and empirically analyze RKL by decomposing the gradients into target and non-target components. While similar decompositions have been explored for FKL in vision settings ( 28 ; 24 ; 6 ) , the optimization dynamics of RKL in large-vocabulary LLM distillation remain largely unexplored. Our analysis reveals a distinct mechanism: non-target gradients consistently push the target logit upward, continuing to increase it even after the student’s target probability matches that of the teacher, which explains both limitations. Motivated by this observation, we propose Diversity-aware RKL (DRKL) , which removes the contribution of non-target gradients to the target logit and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across multiple datasets and model families show that DRKL consistently improves the fidelity–diversity trade-off. Our contributions are summarized as follows: • We show that the empirical advantage of RKL over FKL in LLM distillation arises from its alignment with the structural properties of LLMs. Specifically, under large vocabularies and teacher–student capacity mismatch, RKL simplifies alignment by concentrating learning on dominant modes (in Section 4.1 ). • We theoretically identify key limitations of RKL (in Section 4.2 ). First, RKL creates a shortcut that minimizes the non-target loss without improving alignment over non-target classes (in Proposition 4.1 ). Second, non-target gradients with respect to target logit is consistently negative, which pushes the target logit upward and leads to overconfident and low-diversity predictions (in Proposition 4.2 ). • We propose DRKL to address limitations, which removes the non-target gradients with respect to the target logit and strengthens non-target supervision (in Section 4.3 ). We also present experiments across datasets and model families, demonstrating that DRKL consistently improves performance and fidelity–diversity trade-off (in Section 5 ). 2 Related Works KD for LLM distillation . KD ( 9 ) is widely used to compress LLMs into smaller and more efficient students ( 25 ) . Existing approaches can be broadly categorized into black-box and white-box distillation. Black-box KD relies solely on the outputs of the teacher model. For example, sequence-level KD ( 14 ) trains the student on responses generated by the teacher. In contrast, white-box KD leverages richer teacher signals during training, typically by aligning token-level output distributions between teacher and student. This alignment is commonly achieved by minimizing a KL divergence between their logits during token generation, using either FKL ( 9 ; 13 ) or RKL ( 7 ; 1 ; 12 ) . KL objectives in KD . FKL was originally introduced for KD by 9 and has been widely adopted across vision and language tasks. Recent LLM distillation methods increasingly employ RKL due to its stronger empirical performance ( 7 ; 1 ) . Several works investigate the relationship between FKL and RKL. For example, 26 show that the two objectives converge to the same optimum under sufficient optimization and propose AKL to balance their behavior during early training. Similarly, AB ( 21 ) balances mode-seeking and mean-seeking effects, while SFKL and SRKL ( 15 ) alleviate teacher–student mismatch by interpolating between teacher and student distributions. Another line of work studies distillation objectives through target–non-target decomposition ( 28 ; 24 ; 6 ) , which decomposes FKL into target and non-target components, showing that these terms contribute differently to training dynamics. These methods primarily address the vanishing gradients for non-target classes in vision tasks. In contrast, we leverage this decomposition to analyze the structural limitation of RKL in the large-vocabulary regime of LLM distillation, and uncover that non-target gradients exert persistent upward pressure on the target logit, driving the student toward overconfident and low-diversity predictions. 3 Preliminaries Problem formulation . In this work, we consider the instruction following setting. Let \mathbf{x}=(x_{1},\dots,x_{L}) denote an input sequence of length L , and \mathbf{y}=(y_{1},\dots,y_{T}) denote an output sequence of length T . At decoding step t , given the prefix \mathbf{y}_{<t}:=(y_{1},\dots,y_{t-1})

Read the complete text