An Empirical Study of Knowledge Distillation for Code Understanding Tasks DOI: XXXXXXX.XXXXXXX Conference: the 48th IEEE/ACM International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Software creation a

Ruiqi Wang, Zezhou Yang, Cuiyun Gao, Xin Xia, Qing Liao · 2025

Read An Empirical Study of Knowledge Distillation for Code Understanding Tasks DOI: XXXXXXX.XXXXXXX Conference: the 48th IEEE/ACM International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Software creation a by Ruiqi Wang, Zezhou Yang, Cuiyun Gao, Xin Xia, Qing Liao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15423.txt.

Introduction

Ruiqi Wang and co-authors run a broad test of knowledge distillation for teaching small code models from big ones, in a 2025 study. Big pretrained models read and write code well but are slow and costly to run at scale. Distillation trains a compact student to copy a teacher, yet the method had barely been tried on code tasks.

The team compared two families of distillation, one copying the teacher's answer scores and one copying its inner features. They ran eight student models against two teachers, one general and one code-specific, across three code tasks. Distillation beat plain fine-tuning for students of every size. The code-specific teacher taught best. The newest feature-copying methods let students keep up to 98 percent of teacher skill with only 5 percent of the parameters. Oddly, students shaped like their teacher did not always win.

Catalog record

CREATORRuiqi Wang, Zezhou Yang, Cuiyun Gao, Xin Xia, Qing Liao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study of Knowledge Distillation for Code Understanding Tasks DOI: XXXXXXX.XXXXXXX Conference: the 48th IEEE/ACM International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Software creation and management Ruiqi Wang, Zezhou Yang, Cuiyun Gao, Xin Xia, Qing Liao Abstract . Pre-trained language models (PLMs) have emerged as powerful tools for code understanding. However, deploying these PLMs in large-scale applications faces practical challenges due to their computational intensity and inference latency. Knowledge distillation (KD), a promising model compression and acceleration technique, addresses these limitations by transferring knowledge from large teacher models to compact student models, enabling efficient inference while preserving most of the teacher models’ capabilities. While this technique has shown remarkable success in natural language processing and computer vision domains, its potential for code understanding tasks remains largely underexplored. In this paper, we systematically investigate the effectiveness and usage of KD in code understanding tasks. Our study encompasses two popular types of KD methods, i.e., logit-based and feature-based KD methods, experimenting across eight student models and two teacher PLMs from different domains on three downstream tasks. The experimental results indicate that KD consistently offers notable performance boosts across student models with different sizes compared with standard fine-tuning. Notably, code-specific PLM demonstrates better effectiveness as the teacher model. Among all KD methods, the latest feature-based KD methods exhibit superior performance, enabling student models to retain up to 98% teacher performance with merely 5% parameters. Regarding student architecture, our experiments reveal that similarity with teacher architecture does not necessarily lead to better performance. We further discuss the efficiency and behaviors in the KD process and inference, summarize the implications of findings, and identify promising future directions. 1. Introduction Automated code understanding has emerged as a critical component in software engineering, enabling machines to analyze and process source code to facilitate various downstream tasks. By transforming code into continuous vector spaces that preserve semantic and structural information, these techniques have revolutionized capabilities in clone detection ( Wang et al. 2020a ) , code search ( Shi et al. 2023 ; Liu et al. 2022 ) , vulnerability detection ( Wen et al. 2023 ; Chen et al. 2024 ) , and other software engineering tasks ( Lu et al. 2021 ; Chen et al. 2021b ; Yang et al. 2025 ) . In recent years, pre-trained language models ( Guo et al. 2021 ; Wang et al. 2021b ) have become the dominant paradigm in code understanding tasks, achieving remarkable success across various downstream tasks ( Chen et al. 2022b ) . These models, such as CodeBERT ( Feng et al. 2020 ) and UniXcoder ( Guo et al. 2022 ) , leverage Transformer architectures ( Vaswani et al. 2017a ) trained on large code corpora to capture syntactic and semantic relationships in programming languages, with advanced capabilities in program understanding ( Wang et al. 2022 ) . Furthermore, some general-purpose models like ModernBERT ( Warner et al. 2024 ) are trained for both text and code processing, also demonstrating outstanding code understanding capability. However, these models typically feature hundreds of millions of parameters, making them resource-intensive to deploy in real development environments, leading to notably slower inference. This is particularly problematic since developers often require real-time responsiveness and cannot tolerate delays of seconds or minutes during basic development activities ( Wang et al. 2023 ; Melicher et al. 2021 ) . Consequently, there is growing interest in developing models with strong code understanding capability and minimized computational costs. To balance the performance and efficiency of models, knowledge distillation has emerged as a particularly promising approach. Unlike other compression and acceleration techniques such as pruning ( Michel et al. 2019 ; Frankle and Carbin 2019 ) and quantization ( Wei et al. 2023 ; Tao et al. 2022 ) , knowledge distillation ( Sanh et al. 2019 ; Tian et al. 2019 ; Sun et al. 2019 ; Jiao et al. 2020 ; Li et al. 2021 ; Shi et al. 2022 ; Kim and Rush 2016 ; Shleifer and Rush 2020 ) offers a flexible framework that transfers the knowledge from a large teacher model to a smaller, more efficient student model, enabling practical deployment without considerable performance degradation. This technique allows the student model to learn not just from ground truth but also from the representation space of the teacher model. Knowledge distillation methods can be broadly categorized into two types: logit-based methods, which utilize only the teacher’s predicted probabilities, and feature-based methods, which leverage the teacher’s intermediate feature representations ( Xu and McAuley 2023 ) . While knowledge distillation has demonstrated substantial success in the field of natural language processing (NLP) ( Sanh et al. 2019 ; Jiao et al. 2020 ) and computer vision (CV) ( Tian et al. 2019 ) , it has not been comprehensively explored in code understanding. The lack of relevant research limits the practical deployment of large models for code understanding in resource-constrained environments, or when efficiency and responsiveness are paramount. To mitigate this gap, we systematically investigate the effectiveness of existing knowledge distillation methods on code understanding models. Compared to text and images, code presents unique challenges in understanding tasks due to its distinct structural properties, semantic relationships, and domain-specific characteristics, which may influence the effectiveness of knowledge distillation approaches. Therefore, it is necessary to study several fundamental aspects in this unique domain, including achievable compression ratios, the performance impact of knowledge distillation methods, and optimal distillation settings for code-specific tasks. Guided by these objectives, our empirical study addresses the following research questions (RQs): • RQ1: What is the impact of knowledge distillation on code understanding tasks? We aim to assess whether knowledge distillation can achieve model compression while retaining performance on code understanding tasks. Specifically, we choose defect detection, clone detection, and exception classification in our experiments. We adopt the vanilla logit-based knowledge distillation method on RoBERTa ( Liu et al. 2019 ) student models of five different sizes from 0.6M to 124M, with two teacher models, UniXcoder (a popular code PLM) and ModernBERT (the latest general PLM). We evaluate the effectiveness of knowledge distillation by comparing the performance between standard fine-tuned and distilled student models. • RQ2: How do different knowledge distillation methods affect the model performance for code understanding tasks? Recent advances in knowledge distillation have introduced various techniques to better leverage teacher knowledge and optimize the training process. To explore the impact of different knowledge distillation methods, we evaluate four representative methods across two widely-used model architectures: RoBERTa (Transformer-based) and BiGRU (RNN-based). By examining these methods on student models of varying sizes, we aim to verify their generalizability across architectures and model scales. • RQ3: How do different student models influence the effectiveness of knowledge distillation? While Transformers excel in code and text processing, their high computational demands may affect their efficiency as compressed models. Building upon our investigation of RoBERTa and BiGRU in RQ2, we conduct a deeper analysis of the performance of different student model architectures (Transformer-based and RNN-based). Our experiments provide a more comprehensive overview of the impact of student models on knowledge distillation. We obtain the following findings when answering the RQs: • Knowledge

Read the complete text