Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models

Junjie Yang ∗ 1 , Junhao Song ∗† 2,12 , Xudong Han ∗ 3 , Ziqian Bi ∗ 4,12 , Tianyang Wang 5,12 , Chia Xin Liang 6,12, Xinyuan Song 7,12 , Yichao Zhang 8,12 , Qian Niu 9,12 , Benji Peng 10,12 , Keyu Chen 11,12 , Ming Liu 4,12 · 2025

Read Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models by Junjie Yang ∗ 1 , Junhao Song ∗† 2,12 , Xudong Han ∗ 3 , Ziqian Bi ∗ 4,12 , Tianyang Wang 5,12 , Chia Xin Liang 6,12, Xinyuan Song 7,12 , Yichao Zhang 8,12 , Qian Niu 9,12 , Benji Peng 10,12 , Keyu Chen 11,12 , Ming Liu 4,12 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13825.txt.

Introduction

Shrinking a big AI model into a smaller one without losing skill is the subject of a 2025 survey by Junjie Yang and co-authors. The method, knowledge distillation, trains a compact student model to copy a larger teacher, and it is now used across image and language tasks.

The survey gathers recent twists on the basic idea, such as attention-based methods, block-wise matching of outputs, and methods that split the teacher's knowledge into parts. For language models, these methods cut computing cost and speed up replies while keeping accuracy close to the teacher's. The authors sum up the main findings and open problems, giving researchers and builders a map of where the field is heading.

Catalog record

CREATORJunjie Yang ∗ 1 , Junhao Song ∗† 2,12 , Xudong Han ∗ 3 , Ziqian Bi ∗ 4,12 , Tianyang Wang 5,12 , Chia Xin Liang 6,12, Xinyuan Song 7,12 , Yichao Zhang 8,12 , Qian Niu 9,12 , Benji Peng 10,12 , Keyu Chen 11,12 , Ming Liu 4,12
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13825

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models Thanks: ∗ \ast : Equal contribution. † \dagger : Corresponding author. Junjie Yang ∗ 1 , Junhao Song ∗† 2,12 , Xudong Han ∗ 3 , Ziqian Bi ∗ 4,12 , Tianyang Wang 5,12 , Chia Xin Liang 6,12, Xinyuan Song 7,12 , Yichao Zhang 8,12 , Qian Niu 9,12 , Benji Peng 10,12 , Keyu Chen 11,12 , Ming Liu 4,12 Abstract Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancements in various applications including image classification, object detection, language modeling, text classification, and sentiment analysis. Recent innovations in KD methods, such as attention-based approaches, block-wise logit distillation, and decoupling distillation, have notably improved student model performance. These techniques focus on stimulus complexity, attention mechanisms, and global information capture to optimize knowledge transfer. In addition, KD has proven effective in compressing large language models while preserving accuracy, reducing computational overhead, and improving inference speed. This survey synthesizes the latest literature, highlighting key findings, contributions, and future directions in knowledge distillation to provide insights for researchers and practitioners on its evolving role in artificial intelligence and machine learning. I Introduction Teacher Model Student Model Input Data Hard Labels Knowledge Distillation Soft Labels p_{i}=\frac{\exp(z_{i}/T)}{\sum_{j}\exp(z_{j}/T)} Large, Complex, High Performance Small, Efficient, Fast Inference Fig. 1 : An overview of the Knowledge Distillation (KD) framework, where a large, high-performance teacher model transfers knowledge to a lightweight student model. Soft labels derived from the teacher’s softened logits guide the student’s learning alongside hard labels. Knowledge Distillation (KD) has emerged as a fundamental technique in deep learning, enabling the transfer of knowledge from a complex and pretrained model (the teacher model) to a simpler and smaller model (the student model) ( Hinton et al.,, 2015 ; Gao,, 2023 ) . This approach is effective for model compression, significantly improving their efficiency and making them more suitable for deployment on resource-limited devices. The core idea behind KD is to distill the dark knowledge embedded in the teacher model into a student model, which enhances student performance ( Yu et al.,, 2024 ) . Recent studies have focused on enhancing the effectiveness of KD by developing innovative variants, frameworks, and methodologies, including decoupling logit-based and feature-based knowledge distillation methods ( Zhang et al., 2024c, ) , correlation-aware knowledge distillation ( Zhang et al.,, 2021 ) , and partial-to-whole knowledge distillation ( Abbasi et al.,, 2020 ) . TABLE I : Qualitative Comparison of Teacher and Student Models. Model Type Size Inference Speed Memory Usage Teacher Model Large Slow High Student Model Small Fast Low A common theme among these studies is the emphasis on feature alignment, which is crucial for Knowledge Distillation (KD) ( Yu et al.,, 2024 ) . Aligning the features of teacher and student models enhances the knowledge transfer process, leading to improved model performance. As shown in Figure 1 , another important aspect of KD is the decomposition of knowledge into smaller components, which can facilitate more effective knowledge transfer ( Zhang et al.,, 2021 ) . This approach has been explored in various studies, where methods for decomposing teacher knowledge into subnetworks with increasing channel width have been proposed. For example, recent studies have investigated novel strategies for splitting teacher models into smaller, more manageable components to improve student model efficiency ( Hou and Fan,, 2021 ; Ruffy and Chahal,, 2019 ) . The development of a general framework or model for knowledge distillation (KD) remains an active area of research because it offers a unified perspective on diverse KD techniques and facilitates the systematic analysis and development of new strategies ( Wang et al., 2023a, ) . Significant contributions have been made in this area, with innovative frameworks and methodologies proposed to enhance the efficiency and scalability of KD. Recent studies have explored the applicability of KD across various domains, including natural language processing ( Wang et al.,, 2022 ) and computer vision ( Zhou et al.,, 2020 ) , demonstrating its versatility and broad potential for practical deployment. The complexity of various distillation methods can introduce overhead to the training process ( Zhang et al., 2024c, ) . In addition, the scalability and generalizability of KD techniques are critical considerations for practical applications ( Zhang et al.,, 2021 ) . Furthermore, the choice of teacher-student architecture and the design of the distillation loss function can significantly influence KD performance ( Abbasi et al.,, 2020 ) . To address these challenges, further research is required to develop more efficient and robust KD techniques. In summary, knowledge distillation has become a crucial component of deep learning research, offering a wide range of applications and potential benefits. By examining the key themes and developments in the field, researchers can build on the existing work and push the boundaries of KD. As the field continues to evolve, it is essential to address the challenges and limitations associated with KD, ultimately leading to more efficient, effective, and scalable knowledge transfer techniques ( Feng et al.,, 2023 ) . This paper presents a survey of recent advancements in KD, providing a comprehensive overview of the current state of the field and highlighting its significant contributions, trends, and future directions. II Background and Foundations The concept of knowledge distillation (KD) has gained significant attention in recent years, with numerous studies focusing on improving the effectiveness of this technique ( Ku et al.,, 2020 ; Zhang et al.,, 2021 ; Gao,, 2023 ) . The architecture of the knowledge distillation framework can be visualized, as shown in Figure 2 , which illustrates the key components and their interactions in the distillation process. This framework typically involves a teacher model, which is pretrained on a large dataset, and a student model that learns from the teacher’s output. This process allows the student to mimic the teacher’s behavior and transfer knowledge in a more efficient manner, especially for applications where computational resources are limited. Fig. 2 : Foundations of Knowledge Distillation. The teacher model uses the temperature parameter T to produce softened predictions (soft targets) that capture nuanced information and guide the student model. The student is trained with a loss combining KL divergence between its outputs and the teacher’s soft targets, and cross-entropy with the true labels. Knowledge distillation (KD) involves transferring knowledge from a pre-trained teacher model to a student model, typically with a smaller capacity, to enhance student performance ( Ruffy and Chahal,, 2019 ) . This is typically achieved by optimizing a combined loss function that blends hard-target supervision with a softened probability distribution from the teacher model. \mathcal{L}_{\text{KD}}=(1-\lambda)\cdot\mathcal{L}_{\text{CE}}(\mathbf{y},\mathbf{p}_{s})+\lambda\cdot T^{2}\cdot\mathcal{L}_{\text{KL}}\left(\sigma(\mathbf{z}_{t}/T),\sigma(\mathbf{z}_{s}/T)\right), (1) where \mathcal{L}_{\text{CE}} is the standard cross-entropy loss, \mathcal{L}_{\text{KL}} is the Kullback-Leibler divergence ( Hinton et al.,, 2015 ) , T is the temperature, \lambda balances the two terms, and \sigma(\cdot) denotes the softmax function. This process has been shown to be highly effective in various applications including image classification, natural language processing, and speech recognition. Recent studies have introduced new variants of knowledge distillation (KD), such as teaching-assistant distillation, curriculum distillation, mask distillation, and decoupling distillation, which aim to improve the performance of KD

Read the complete text