Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation
Md. Samiul Alim, Sharjil Khan, Amrijit Biswas, Fuad Rahman, Shafin Rahman, Nabeel Mohammed · 2025
Read Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation by Md. Samiul Alim, Sharjil Khan, Amrijit Biswas, Fuad Rahman, Shafin Rahman, Nabeel Mohammed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16653.txt.
Introduction
Md. Samiul Alim and five co-authors prune neural networks in one shot, guided by a teacher model, instead of the usual slow cycle of training, pruning, and retraining. Their trick is to let the teacher shape the importance scores themselves, so the weights kept are those that matter both for the task and for knowledge transfer.
After the one-shot global pruning, sparsity-aware retraining, with or without distillation, restores accuracy without reviving pruned connections. On CIFAR-10, CIFAR-100, and TinyImageNet the method reached high sparsity with minimal accuracy loss. It beat strong baselines such as EPG and EPSD at high sparsity and costs far less computation than iterative schemes, making it attractive for deploying models on constrained hardware.
Catalog record
| CREATOR | Md. Samiul Alim, Sharjil Khan, Amrijit Biswas, Fuad Rahman, Shafin Rahman, Nabeel Mohammed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation Md. Samiul Alim, Sharjil Khan, Amrijit Biswas, Fuad Rahman, Shafin Rahman, Nabeel Mohammed Abstract Unstructured pruning remains a powerful strategy for compressing deep neural networks, yet it often demands iterative train–prune–retrain cycles, resulting in significant computational overhead. To address this challenge, we introduce a novel teacher-guided pruning framework that tightly integrates Knowledge Distillation (KD) with importance score estimation. Unlike prior approaches that apply KD as a post-pruning recovery step, our method leverages gradient signals informed by the teacher during importance score calculation to identify and retain parameters most critical for both task performance and knowledge transfer. Our method facilitates a one-shot global pruning strategy that efficiently eliminates redundant weights while preserving essential representations. After pruning, we employ sparsity-aware retraining with and without KD to recover accuracy without reactivating pruned connections. Comprehensive experiments across multiple image classification benchmarks, including CIFAR-10, CIFAR-100, and TinyImageNet, demonstrate that our method consistently achieves high sparsity levels with minimal performance degradation. Notably, our approach consistently outperforms state-of-the-art baselines such as EPG, and EPSD at high sparsity levels, while offering a more computationally efficient alternative to iterative pruning schemes like COLT. The proposed framework offers a computation-efficient, performance-preserving solution well-suited for deployment in resource-constrained environments. I Introduction Fig. 1: Overview of accuracy comparison between our method and existing approaches across sparsity levels (50%, 75%, 90%, 94%, 98%). In the radar plot, a larger enclosed area indicates higher accuracy, where Our Method consistently achieves superior performance on CIFAR-10 datasets, clearly outperforming competing baselines. Detailed results including CIFAR-10 and other datasets are presented in the Results section. Deep Neural Networks (DNNs), particularly Convolutional Neural Networks (CNNs), have achieved state-of-the-art performance across numerous computer vision tasks such as image classification [ 13 , 17 ] , object detection, and semantic segmentation [ 4 ] . However, these performance gains often come with a high computational and memory cost [ 17 ] , limiting their deployment on resource-constrained environments such as mobile devices and edge platforms. To address this, the community has explored various model compression techniques aimed at reducing model complexity while retaining performance. Common approaches include parameter quantization [ 7 ] , efficient architecture design [ 13 ] , knowledge distillation (KD) [ 10 ] , and network pruning [ 8 , 2 ] . Among these, network pruning stands out for its ability to significantly reduce parameter count by removing redundant weights, filters, or neurons. It is broadly categorized into structured pruning [ 19 , 22 ] , which removes entire channels or filters for hardware efficiency, and unstructured pruning [ 9 , 6 ] , which eliminates individual weights and can achieve higher sparsity levels. While structured pruning improves practical deployment, it often sacrifices fine-grained control and performance. Unstructured pruning, in contrast, allows finer sparsity control and higher compression ratios but typically relies on less informative heuristics such as magnitude-based criteria [ 9 ] -to determine parameter importance. Several limitations persist in the current landscape of unstructured pruning. First, importance estimation is often based on heuristics (e.g., weight magnitude) that fail to capture dynamic learning signals. This limitation is evident in approaches like the Lottery Ticket Hypothesis (LTH) [ 6 ] , which require multiple train-prune-retrain iterations to isolate performant subnetworks, resulting in excessive computational cost. Second, while gradient-based criteria [ 18 , 23 ] offer a more principled alternative by leveraging sensitivity, they often rely on noisy single-step gradients and are detached from any auxiliary supervision such as that from a teacher network. Third, knowledge distillation (KD) - a powerful framework to transfer knowledge from a large teacher to a smaller student - has been predominantly used as a post-pruning recovery tool [ 24 , 1 ] , rather than being integrated into the pruning process itself. Consequently, pruning decisions are made without the benefit of the teacher’s informative soft targets. While EPSD [ 3 ] addresses this challenge through self-distillation-aware early pruning, our approach extends this by using a pretrained teacher to actively guide the pruning process via task-aligned gradients. Our goal is to address these limitations by introducing a teacher-guided unstructured pruning framework that leverages both gradient sensitivity and knowledge distillation in a unified pipeline. Unlike iterative methods such as LTH [ 6 ] and COLT [ 11 ] , which require costly train-prune-retrain cycles, our approach enables efficient one-shot global pruning. While prior efforts like EPSD [ 3 ] incorporate self-distillation to improve early-stage pruning, they lack explicit teacher supervision during importance estimation. In contrast, we incorporate the teacher’s guidance directly into the pruning signal through a distillation-aware loss function specifically, Context-Aware Kullback Leibler Divergence (CA-KLD) [ 21 ] augmented with logit normalization [ 27 ] for stable optimization. Gradient signals informed by this combined loss are aggregated using an exponential moving average with bias correction and used to rank parameter importance. The resulting scores enable aggressive one-shot pruning, followed by sparsity-aware retraining either with or without KD while strictly preserving the pruned structure. The main contributions of our work are: • A novel teacher-guided gradient importance metric that utilizes gradients derived from both task loss and an advanced KD loss (CA-KLD). • A demonstration of integrating advanced KD not just for post-pruning recovery, but as an active component guiding the identification of critical parameters during the importance score calculation. • Our One-Shot Global Pruning method minimizes computational cost compared to iterative pruning methods such as Lottery Ticket Hypothesis [ 6 ] and COLT [ 11 ] . Furthermore, our method outperforms six different methods evaluated in the EPSD [ 3 ] , including CS-KD Simple, CS-KD EPSD, PS-KD Simple,PS-KD EPSD, DLB Simple, and DLB EPSD, as well as the EGP method [ 20 ] . • Extensive empirical evaluation on benchmark datasets (CIFAR-10, CIFAR-100, TinyImageNet) using standard architectures (ResNet18, ResNet34), demonstrating competitive performance compared to other methods. II Related Work Deep neural networks, particularly convolutional neural networks (CNNs), have achieved remarkable success across various domains, but their growing computational and memory demands pose challenges for deployment on resource-constrained devices [ 17 ] . This has motivated extensive research into model compression techniques, including parameter quantization [ 7 ] , knowledge distillation (KD) [ 10 ] , designing compact architectures [ 13 ] , and network pruning [ 8 , 2 ] . Our work focuses on pruning enhanced by KD. Network Pruning: Network pruning reduces model complexity by removing redundant weights, neurons, or filters while minimizing performance loss. Unstructured pruning eliminates individual weights, often based on magnitude [ 9 , 6 ] , resulting in sparse networks that require specialized hardware or libraries for efficient inference. The Lottery Ticket Hypothesis (LTH) [ 6 ] suggests that dense networks contain sparse "winning ticket" subnetworks that can match the original model’s performance when trained from early initialization. Structured pruning , on the other hand, removes entire filters, channels, or layers [ 19 , 22 ] , producing smaller dense networks compatible with standard hardware, though potentially more detrimental to accuracy, especially in architectures like ResNets [ 1 ] . Pruning criteria include magnitude-based methods [ 9 , 6 ] , activation-based metrics such as APoZ [ 12 ] or channel entropy [ 22 , 20 ] ,