Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation
Muhammad Haseeb Aslam (✉), Clara Martinez, Marco Pedersoli, Alessandro Lameiras Koerich, Ali Etemad, Eric Granger · 2025
Read Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation by Muhammad Haseeb Aslam (✉), Clara Martinez, Marco Pedersoli, Alessandro Lameiras Koerich, Ali Etemad, Eric Granger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14307.txt.
Introduction
Muhammad Haseeb Aslam and co-authors present a training method from two thousand twenty-five for compact models that learn from a single model's many faces. Distilling a teacher into a student can beat the teacher, and ensembles help too, but training many teachers is costly for wearables and other tight settings.
Their idea trains just one model, then uses dropout at teaching time to spawn many varied teacher views. Those views are noisy, so the student itself guides which ones to trust: at each step, the student's current state filters and weights the teacher signals toward task-relevant ones. The authors test the method on real-world emotion and biosignal datasets from a public archive. The paper targets settings where storage and delay budgets are strict.
Catalog record
| CREATOR | Muhammad Haseeb Aslam (✉), Clara Martinez, Marco Pedersoli, Alessandro Lameiras Koerich, Ali Etemad, Eric Granger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation Muhammad Haseeb Aslam (✉), Clara Martinez, Marco Pedersoli, Alessandro Lameiras Koerich, Ali Etemad, Eric Granger Abstract Advances in self-distillation have shown that when knowledge is distilled from a teacher to a student using the same deep learning (DL) model, student performance can surpass the teacher, particularly when the model is over-parameterized and the teacher is trained with early stopping. Alternatively, ensemble learning also improves performance, although training, storing, and deploying multiple DL models becomes impractical as the number of models grows. Even distilling a deep ensemble to a single student model or weight averaging methods first requires training of multiple teacher models and does not fully leverage the inherent stochasticity for generating and distilling diversity in DL models. These constraints are particularly prohibitive in resource-constrained or latency-sensitive applications on, e.g., wearable devices. This paper proposes to train only one model and generate multiple diverse teacher representations using distillation-time dropout . However, generating these representations stochastically leads to noisy representations that are misaligned with the learned task. To overcome this problem, a novel stochastic self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only, using student-guided knowledge distillation. The student representation at each distillation step is used to guide the distillation process. Experimental results 1 1 1 Code and supplementary available at: https://github.com/haseebaslam95/SSD on real-world affective computing, wearable/biosignal (UCR Archive), HAR, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods without increasing the model size at both training and testing time. It incurs negligible computational complexity compared to ensemble learning and weight averaging methods. 1 Introduction Wearable technology has many applications, primarily in healthcare monitoring, such as activity and exercise tracking, sleep analysis, stress detection, and fall detection. It also includes applications like chronic disease management, personalized health insights, and human behavior and physiology research by continuously tracking metrics like heart rate, steps taken, body temperature, and movement patterns over time. Time-series signals such as electrocardiogram (ECG), respiration rate, and other biosignals are often multi-dimensional, noisy, and collected in real time from resource-constrained devices. These signals require efficient processing methods that balance accuracy with computational efficiency. Cumbersome methods for performance boosting are less effective for this application. Knowledge distillation (KD) is typically used for transferring knowledge from a large, well-trained teacher model to a more compact student model for deployment, thereby enhancing the latter’s accuracy without incurring significant computational costs [ 12 ] . Self-distillation is a specialized case in KD, where the teacher and student have the same DL architecture, and the student typically surpasses the teacher’s performance particularly where the model is over-parameterized i.e., has sufficient capacity and the teacher is trained with early-stopping. This increase in performance is typically associated with the fact that, with DL models, the teacher and the student have learned separate discriminative features, and self-distillation implicitly ensembles the two models [ 1 ] . Diversity in the feature space is a critical factor that enhances the robustness and accuracy of machine learning models. Diverse representations provide a comprehensive understanding of the input data, mitigating overfitting and improving generalization across various tasks [ 9 ] [ 17 ] . Approaches for ensemble learning leverage the independent training of multiple diverse models to learn more robust decision boundaries, leading to significant improvements in predictive accuracy. Despite these advantages, deploying deep ensembles introduces substantial computational and storage overhead, as each model in the ensemble requires independent training, parameter storage, and inference pipelines. These constraints are particularly prohibitive in resource-constrained embedded systems as employed in wearable applications. State-of-the-art (SOTA) approaches [ 1 ] that distill diverse ensemble-based representations involve the cumbersome process of training the teacher model multiple times or utilizing complex ensemble learning methods to generate a pool of diverse teacher models for effective knowledge transfer. These methods are computationally intensive and may not fully leverage the potential of stochasticity inherent in DL models for generating diversity. This paper introduces a KD training strategy called Stochastic Self-Distillation (SSD) to capitalize on distillation-time dropout, thereby inducing stochasticity in a single, pre-trained teacher model. SSD generates multiple stochastic feature representations, effectively simulating a diverse ensemble of DL models without requiring extensive teacher re-training. This technique aligns with the principles of Monte Carlo dropout [ 11 ] . Moreover, a Student-Guided Knowledge Distillation (SGKD) is introduced to distill the most relevant knowledge (or filter out noisy representations) to the student model using student-guided attention. This mechanism allows the student to selectively focus on the most informative representations within the teacher’s output space, facilitating a more efficient and targeted knowledge transfer. Subsequently, feature-level KD is employed to align the student’s feature representations with the filtered and attention-weighted teacher feature representation. The main contributions of this paper are summarized as follows. (1) We propose SSD, a novel distillation-time dropout strategy to generate diverse stochastic representations from a single, pre-trained teacher model. (2) Within SSD, a novel SGKD mechanism enables the student model to selectively distill knowledge from the most informative teacher representations. Feature-based KD is used to align the student’s internal feature space with the teacher, promoting a more granular knowledge transfer. (3) Our extensive experiments on challenging affective computing benchmark datasets (Biovid Pain and StressID), biosignal/wearable datasets (from the UCR Archive), the HAR dataset, and benchmark image classification datasets (CIFAR-10 and CIFAR-100) show that our SSD training strategy allows training models that can achieve SOTA performance while maintaining computational efficiency. 2 Related Work Knowledge Distillation. Originally introduced by [ 5 ] [ 12 ] , the KD domain has evolved with several refinements in its application and architecture. Romero et al. [ 29 ] introduced the concept of distilling from feature representations instead of logits. The idea of transferring the attention maps from the teacher model to the student model was studied by Zagoruyko and Komodakis [ 41 ] . Relational KD proposed by Park et al. [ 27 ] studied the benefits of utilizing structural information for more fine-grained KD. The KD domain was extended to multi-task, semi-supervised, and unsupervised learning by Lopez et al. [ 23 ] . KD has also been studied in multimodal systems, particularly with applications like cross-modal KD [ 30 ] privileged KD [ 3 ] , federated learning [ 21 ] , are a few examples of the widespread application of KD in real-world systems. Deep Ensembles and Model Soups. Ensembling methods improve predictive performance, generalization in neural networks, and uncertainty estimation. Deep ensemble is a simple yet effective technique where a simple aggregation of independently trained models harnesses the diversity, leading to better performance than each model. Lakshminarayanan et al. [ 17 ] demonstrated the effectiveness of deep ensembles for uncertainty estimation, showing that they outperform many Bayesian approaches in terms of both calibration and robustness. Deep theoretical insights on ensemble diversity were provided by Fort et al. [ 9 ] . Moreover, Ovadia et al. [ 26 ] highlight the advantages of deep ensembles in handling distributional shifts, reinforcing their utility in real-world scenarios. Despite their advantages, deep ensembles are computationally expensive, requiring the training and storage of multiple models, which motivates research into alternative methods that capture similar benefits