Rethinking the Role of Temperature in Large Language Model Distillation

Hoang-Chau Luong Lingwei Chen · 2026

Read Rethinking the Role of Temperature in Large Language Model Distillation by Hoang-Chau Luong Lingwei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00306.txt.

Introduction

Luong and Chen rethink the heat dial in how big models teach small ones. In distillation, a teacher's soft picks guide a student net. A dial named temperature softens those picks. Past picks twixt two gap gauges left that dial out of view.

Their reads show the dial acts in a lopsided way on the two gauges. It feeds rich side cues to the forward gauge. For the reverse gauge, it just scales step sizes. At low heat, the reverse gauge wins the race. At high heat, the forward gauge wins across guide tests. Heat helps a broad kin of teach goals too. Plain gap tools with tuned heat tie fresh top tools.

Catalog record

CREATORHoang-Chau Luong Lingwei Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking the Role of Temperature in Large Language Model Distillation Hoang-Chau Luong Lingwei Chen Abstract Reverse Kullback–Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature τ \tau , overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from τ \tau scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at τ = 1 \tau=1 , FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches. 1 Introduction Knowledge distillation (KD) ( 9 ) is a widely used paradigm for model compression that transfers knowledge from a large teacher model to a smaller student model ( 20 ; 4 ; 7 ) . A central component of classical KD is the distillation temperature \tau , which softens teacher distribution and reveals informative “dark knowledge” beyond the top prediction ( 9 ; 25 ; 33 ) . By redistributing probability mass from dominant classes to lower-probability alternatives, KD temperature prevents the distillation signal from collapsing onto the teacher’s top prediction, and provides the student with richer inter-class relational information. Although temperature has been extensively studied in vision-based KD 2 ; 15 ; 23 ; 24 , it is often omitted in LLM distillation. Early sequence-level KD work for language tasks ( 13 ) reported that \tau=1 achieved the best empirical performance and therefore adopted the default softmax distribution without further analyzing the role of temperature. Subsequent LLM distillation methods similarly formulate and evaluate their distillation frameworks under the default \tau=1 setting ( 8 ; 1 ; 14 ; 26 ; 22 ) , implicitly removing temperature from the design space of KL-based distillation objectives. This convention directly affects how recent work compares forward KL (FKL) and reverse KL (RKL) objectives, where the commonly reported superiority of RKL is largely established through evaluations conducted at \tau=1 ( 8 ; 14 ; 30 ) . Unlike RKL, whose optimization is weighted by the student distribution, FKL directly weights each sample loss by the teacher probabilities. As a result, low-entropy teacher soft targets suppress the non-target probability mass that FKL relies on, making it appear inferior not due to an inherent limitation of the objective itself, but because it is evaluated under insufficiently softened teacher distributions. This raises an important question: Is RKL intrinsically superior for LLM distillation or has FKL been systematically underestimated? 1.1 Related Work Temperature in KD . Given teacher logits z^{t} and student logits z^{s} , temperature-scaled distributions are defined as p^{\tau}=\mathrm{softmax}(z^{t}/\tau),\ q^{\tau}=\mathrm{softmax}(z^{s}/\tau), (1) where \tau\geq 1 . Increasing \tau smooths the distribution by reducing probability mass on dominant tokens and amplifying lower-probability alternatives. This smoothing has been widely used in vision-based distillation ( 9 ; 33 ; 15 ; 12 ; 23 ; 6 ; 29 ) , as well as in early NLP distillation methods such as Seq-KD ( 13 ) , DistilBERT ( 21 ) , and TinyBERT ( 11 ) . In LLM distillation, however, most methods omit KD temperature and evaluate objectives at \tau=1 ( 8 ; 14 ; 30 ; 22 ) . Although recent work has revisited temperature for FKL ( 22 ) or proposed adaptive temperature strategies ( 31 ; 18 ) , its objective-dependent effect on FKL and RKL remains poorly understood. KL objectives for LLM distillation . FKL and RKL are two standard distribution-matching objectives. FKL minimizes D_{\mathrm{KL}}(p^{\tau}\|q^{\tau}) , encouraging the student to cover the teacher distribution, while RKL minimizes D_{\mathrm{KL}}(q^{\tau}\|p^{\tau}) and is associated with mode-seeking behavior. Recent LLM distillation methods favor RKL due to its strong empirical performance ( 8 ; 14 ) , leading to variants such as AKL ( 30 ) , AB-KD ( 26 ) , SFKL/SRKL ( 14 ) , and DRKL ( 17 ) . However, these comparisons are typically conducted under \tau=1 setting, leaving the interaction between temperature and KL objective design underexplored. 1.2 Contributions We revisit the role of temperature in LLM distillation and address three key questions: RQ1: How does temperature distinguish FKL from RKL? We theoretically show that both FKL and RKL converge to the same logit-matching behavior in the high-temperature regime. At practical temperatures, KD temperature reshapes the teacher soft targets and enriches the non-target knowledge transfer for FKL, while primarily rescaling RKL gradients. Thus, KD temperature changes what FKL learns, but mainly changes how RKL learns. RQ2: When does FKL outperform RKL? We demonstrate that the FKL–RKL comparison is highly temperature-dependent. While RKL often performs better at \tau=1 , FKL benefits substantially more from softened teacher targets and consistently surpasses RKL at higher KD temperatures. This finding challenges the common view that RKL is intrinsically superior for LLM distillation. RQ3: Does temperature broadly improve KL-based distillation objectives? We further show that KD temperature improves not only FKL, but also a broader family of KL-based objectives, including Sym-KL, JS, SFKL, and AKL. With appropriate KD temperature, these simple objectives achieve performance competitive with existing LLM distillation methods. This suggests that KD temperature is not merely a correction for FKL, but a general objective-dependent factor for designing and fairly comparing LLM distillation methods. (a) GPT-2 XL \to GPT-2 Base (b) GPT-2 XL \to GPT-2 Medium (c) OPT 6.7B \to OPT 1.3B Figure 1: Effect of temperature on FKL and RKL across model scales. Temperature consistently improves FKL while providing limited benefit for RKL, leading to a reversal in their performance at higher temperatures. 2 How Does Temperature Distinguish FKL from RKL? This section theoretically analyzes the behavior of FKL and RKL under KD temperature, which shows that the two objectives become equivalent in the high-temperature regime, but diverge substantially at low temperatures, explaining why FKL can outperform RKL in LLM distillation. Let z^{t},z^{s}\in\mathbb{R}^{V} denote the teacher and student logits over a vocabulary of size V . The temperature-scaled teacher and student distributions, denoted by p^{\tau} and q^{\tau} , are defined in Eq. ( 1 ). We study \mathcal{L}_{\mathrm{FKL}}=D_{\mathrm{KL}}(p^{\tau}\|q^{\tau})=\sum_{i=1}^{V}p_{i}^{\tau}\log(p_{i}^{\tau}/q_{i}^{\tau}) and \mathcal{L}_{\mathrm{RKL}}=D_{\mathrm{KL}}(q^{\tau}\|p^{\tau}) . Their gradients with respect to student logit z_{i}^{s} are \displaystyle\nabla_{z_{i}^{s}}\mathcal{L}_{\mathrm{FKL}} \displaystyle=\frac{1}{\tau}(q_{i}^{\tau}-p_{i}^{\tau}), (2) \displaystyle\nabla_{z_{i}^{s}}\mathcal{L}_{\mathrm{RKL}} \displaystyle=\frac{1}{\tau}q_{i}^{\tau}\left(\log\frac{q_{i}^{\tau}}{p_{i}^{\tau}}-\mathcal{L}_{\mathrm{RKL}}\right). (3) Proposition 2.1 . Assume the teacher and student logits are centered, i.e., \sum_{i}z_{i}^{t}=\sum_{i}z_{i}^{s}=0 . As \tau\to\infty , FKL and RKL gradients satisfy \nabla_{z_{i}^{s}}\mathcal{L}_{\mathrm{FKL}}=\nabla_{z_{i}^{s}}\mathcal{L}_{\mathrm{RKL}}=\frac{1}{V\tau^{2}}(z_{i}^{s}-z_{i}^{t})+O(\tau^{-3}). High-temperature regime: FKL and RKL become logit matching . Recent studies ( 30 ; 17 ) suggest that FKL and RKL reach the same solution under idealized assumptions, such as sufficient student capacity and exact global optimization. Proposition 2.1 (proof is provided in Appendix B ) shows that temperature alone is sufficient to recover this equivalence: as \tau\to\infty , both objectives share the same gradient and reduce to teacher–student logit matching. Thus, the practical distinction between FKL and RKL must come from their low-temperature regime. Low-temperature

Read the complete text