Distillation of Large Language Models via Concrete Score Matching

Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon · 2025

Read Distillation of Large Language Models via Concrete Score Matching by Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25837.txt.

Introduction

Yeongmin Kim and four co-authors propose a new way to shrink large language models in this two thousand twenty-five paper. Big models perform well but cost a lot to run. Knowledge distillation trains a small student model to copy a large teacher. Most methods match the two models' output probabilities. That matching blurs fine detail in the teacher's raw scores, called logits.

Their method, Concrete Score Distillation, matches score differences directly instead. It avoids both the blur and a flaw in direct logit copying. The authors also tame the training cost and instability that plagued earlier score matching for text. Both mode-seeking and mode-covering versions are given. The work targets teams that need capable models cheap enough to deploy widely.

Catalog record

CREATORYeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distillation of Large Language Models via Concrete Score Matching Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon Abstract Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Existing KD objectives typically match student and teacher probabilities via softmax, which blurs valuable logit information. While direct logit distillation (DLD) mitigates softmax smoothing, it fails to account for logit shift invariance, thereby restricting the solution space. We propose Concrete Score Distillation (CSD), a discrete score-matching objective that overcomes both softmax-induced smoothing and restrictions on the optimal solution set. We resolve the training instability and quadratic complexity of discrete score-matching in autoregressive LLMs, and the resulting CSD objective aligns relative logit differences across all vocabulary pairs between student and teacher with flexible weighting. We provide both mode-seeking and mode-covering instances within our framework and evaluate CSD on task-agnostic instruction-following and task-specific distillation using GPT-2-1.5B, OpenLLaMA-7B, and GEMMA-7B-IT. Experiments show that CSD consistently surpasses recent KD objectives, achieves favorable fidelity–diversity trade-offs, and yields complementary gains when combined with on-policy techniques, demonstrating its scalability and effectiveness for LLM distillation. 1 Introduction Large language models (LLMs) have demonstrated remarkable generative capabilities across a wide range of tasks ( Achiam et al. 2023 ; Dubey et al. 2024 ; Liu et al. 2024 ; Comanici et al. 2025 ) . Such progress has been primarily driven by the vast amount of training data and the unprecedented scale of model parameters ( Kaplan et al. 2020 ) . However, when deploying such LLMs in real-world applications, the recurring inference cost becomes prohibitively expensive. Consequently, research into reducing the parameter size of LLMs while preserving performance has become particularly crucial for enabling efficient inference. In this context, knowledge distillation (KD) ( Hinton et al. 2015 ) has emerged as a promising approach for LLMs, as it allows a smaller student model to inherit the capabilities of a large teacher model, thereby enabling more efficient inference. The common paradigm in KD for LLMs is to align the per-token probability distributions of the student with those of the teacher. Kullback–Leibler (KL) divergence was initially the most widely adopted objective, and the search for more effective probability matching losses has since become a central topic of research. Alternative objectives have been proposed within the framework of f -divergence ( Wen et al. 2023 ; Gu et al. 2024 ; Agarwal et al. 2024 ) , as well as its smoothed variants ( Ko et al. 2024 ; Shing et al. 2025 ; Ko et al. 2025 ) . However, existing distillation losses primarily targeted the estimated probabilities obtained through the softmax transformation, instead of directly utilizing the raw neural network outputs (logits) from either the teacher or the student. As illustrated in Figure 1(b) , even when the teacher’s logit values differ substantially, their corresponding probability values can be nearly indistinguishable. Such smoothing hinders the student from faithfully capturing the teacher’s knowledge, a challenge further exacerbated in modern LLMs with large vocabularies, where most tokens are assigned near-zero probabilities (See Figure 1(a) ). In traditional KD, direct logit distillation (DLD) ( Ba & Caruana 2014 ; Urban et al. 2017 ) has been proposed as an alternative strategy, with advantages in generalization capability and in removing the softmax smoothing ( Kim et al. 2021 ) . However, such approaches have not been thoroughly explored in the context of LLMs. This paper identifies a key drawback of DLD: its restriction on the optimal solution set as described in Figure 1(c) . Considering the softmax activation in inference, it is sufficient for the teacher’s and student’s logits to agree up to an additive constant, but the previous solutions of DLD fail to accommodate such an acceptable slack constant, a.k.a. logit shift invariance . Such a restriction on the solution set may hinder the discovery of optimal solutions in distillation, particularly when the teacher and student models have a large capacity gap, as is often the case with LLMs. Therefore, the goal of this paper is to establish a design space of distillation losses that overcome both the softmax-induced smoothing of teacher knowledge and the restriction on the solution set. This paper adopts the idea from energy-based models ( Song & Kingma 2021 ) , which design objectives that avoid the constraint of probabilistic models (sum-to-one) by using the score-matching objective ( Hyvärinen & Dayan 2005 ) . We propose Concrete Score Distillation (CSD), a discrete form of the score-matching objective ( Meng et al. 2022 ) adapted for autoregressive LLM distillation. We address training instability and computational overhead arising when applying the score-matching objective to LLMs, and provide theoretical guarantees of optimality, showing that its solution set is broader than that of DLD from both theoretical and empirical perspectives. The resulting objective reduces to matching the relative logit differences across all pairs of vocabulary items between the student and teacher, while allowing flexible weighting across all vocabulary pairs in linear time with respect to vocabulary size. Furthermore, we present instances within our framework that exhibit both mode-seeking and mode-covering properties. In our experiments, we conducted both task-agnostic instruction-following distillation and task-specific distillations (summarization, mathematics, and translation) using GPT-2 ( Radford et al. 2019 ) , OpenLLaMA ( Geng & Liu 2023 ) , and GEMMA ( Team et al. 2024 ) backbones. The proposed CSD consistently outperformed recent probability-matching objectives as well as direct logit distillation. By appropriately choosing weighting functions, we further demonstrated that our method resides on the frontier of the diversity–fidelity trade-off. Finally, we observed complementary performance gains when integrating our loss with on-policy techniques. (a) Probability value statistics. (b) Softmax hides teacher’s knowledge. (c) Optimal solution sets. Figure 1: Motivation for logit-level distillation and limitations of prior work. (a) Statistics of per-token probabilities for every vocabulary for 16 input–output sequences from the teacher model ( GPT-2-1.5B ). The probabilities are highly sparse, with only 0.0023% being greater than 0.01. (b) Despite large differences in logits (e.g., [-1,-4,4] vs. [1,-9,6] ), softmax yields nearly identical probabilities and gradients. (c) Prior direct logit distillation restricts the solution set. 2 Preliminaries 2.1 Knowledge Distillation of Large Language models We consider autoregressive large language models (LLMs), consisting of a teacher p_{T} and a student q_{\theta} , where the student is a smaller and more efficient model. Given an input context {\mathbf{c}} , the student generates an output sequence {\mathbf{y}}=(y_{1},y_{2},\dots,y_{L}) with probability q_{\theta}({\mathbf{y}}|{\mathbf{c}})=\prod_{t=1}^{L}{q_{\theta}(y_{t}|{\mathbf{c}},{\mathbf{y}}_{<t})} , where L denotes the sequence length, and the teacher’s probability is defined analogously. Each token y_{t} is drawn from the fixed vocabulary set \mathcal{V}:=\{v_{1},v_{2},...\} . As in prior works ( Lin et al. 2020 ; Ko et al. 2024 ) , we assume the teacher and student share the same vocabulary set. To compute the token probability q_{\theta}(y_{t}|{\mathbf{c}},{\mathbf{y}}_{<t}) , an LLM typically adopts a parametric function f_{\theta}:{\mathcal{V}}^{|{\mathbf{c}}|}\times{\mathcal{V}}^{t-1}\rightarrow\mathbb{R}^{|{\mathcal{V}}|} , which maps the input ({\mathbf{c}},{\mathbf{y}}_{<t}) to a logit vector f_{\theta}({\mathbf{c}},{\mathbf{y}}_{<t})\in\mathbb{R}^{|{\mathcal{V}}|} . The logit corresponding to token y_{t} is denoted by f_{\theta}({\mathbf{c}},{\mathbf{y}}_{<t})[y_{t}] . For brevity of notation, the explicit input arguments of the function f_{\theta} will be omitted hereafter. Let f_{T} be the parametric function of the teacher. Accordingly, the probability of each token is calculated through the following softmax transformation: q_{\theta}(y_{t}|{\mathbf{c}},{\mathbf{y}}_{<t})=\frac{\text{exp}(f_{\theta}[y_{t}])}{\sum_{x\in\mathcal{V}}\text{exp}(f_{\theta}[x]))},\quad\quad

Read the complete text