Enriching Knowledge Distillation with Cross-Modal Teacher Fusion

Amir M. Mansourian, Amir Mohammad Babaei, Shohreh Kasaei · 2025

Read Enriching Knowledge Distillation with Cross-Modal Teacher Fusion by Amir M. Mansourian, Amir Mohammad Babaei, Shohreh Kasaei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09286.txt.

Introduction

Amir M. Mansourian and two co-authors enrich knowledge distillation. Their student model learns from two kinds of teachers at once. Distillation compresses expert models into a small student, and multi-teacher versions usually rely on visual information alone. Their method adds CLIP, a vision-language model, as a second source of supervision.

The framework fuses logits and features from a conventional teacher with CLIP's, guided by multiple text prompts. The fused teacher makes more confident correct predictions and fewer confidently wrong ones. It also produces more meaningful probabilities for non-target classes, improving consistency across categories. Across benchmarks, their RichKD beats most existing baselines and holds up better under distribution shifts and corrupted inputs.

Catalog record

CREATORAmir M. Mansourian, Amir Mohammad Babaei, Shohreh Kasaei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enriching Knowledge Distillation with Cross-Modal Teacher Fusion Amir M. Mansourian, Amir Mohammad Babaei, Shohreh Kasaei Abstract Multi-teacher knowledge distillation (KD), a more effective technique than traditional single-teacher methods, transfers knowledge from expert teachers to a compact student model using logit or feature matching. However, most existing approaches lack knowledge diversity, as they rely solely on unimodal visual information, overlooking the potential of cross-modal representations. In this work, we explore the use of CLIP’s vision–language knowledge as a complementary source of supervision for KD, an area that remains largely underexplored. We propose a simple yet effective framework that fuses the logits and features of a conventional teacher with those from CLIP. By incorporating CLIP’s multi-prompt textual guidance, the fused supervision captures both dataset-specific and semantically enriched visual cues. Beyond accuracy, analysis shows that the fused teacher yields more confident and reliable predictions, significantly increasing confident-correct cases while reducing confidently wrong ones. Moreover, fusion with CLIP refines the entire logit distribution, producing semantically meaningful probabilities for non-target classes, thereby improving inter-class consistency and distillation quality. Despite its simplicity, the proposed method, En Rich ing K nowledge D istillation (RichKD), consistently outperforms most of existing baselines across multiple benchmarks and exhibits stronger robustness under distribution shifts and input corruptions. 1 Introduction Knowledge Distillation (KD) 8 ; 28 has become one of the most effective strategies for compressing deep neural networks, enabling lightweight student models to inherit the representational power of larger teachers. By transferring knowledge from a well-trained teacher through logit or feature alignment, KD has achieved success across image recognition, object detection, and many other domains. Despite these advances, most existing KD methods remain strictly unimodal , relying solely on visual cues learned from the target dataset. As a result, the distilled knowledge often inherits the teacher’s task-specific biases and lacks the semantic diversity necessary for broader generalization. Figure 1 : Impact of cross-modal teacher fusion on CIFAR-100. (a) Effect of perturbing the logits of the conventional teacher with CLIP’s logits across four categories, considering cases where the teacher’s predictions are correct/incorrect and certain/uncertain. (b) Effect of fusion with CLIP for two sample cases: when the teacher is incorrect, and when the teacher is correct but uncertain. In contrast, large-scale vision–language models such as CLIP 35 capture rich cross-modal knowledge by jointly learning from hundreds of millions of image–text pairs. CLIP’s training paradigm grounds visual concepts in linguistic semantics, enabling zero-shot classification and robust generalization across diverse datasets. However, while CLIP has revolutionized transfer and retrieval tasks, its potential as a cross-modal teacher for knowledge distillation remains largely underexplored 57 . Integrating CLIP’s multimodal understanding into a KD framework presents a promising avenue for transferring both dataset-specific and semantically enriched knowledge to compact visual students. This paper introduces En Rich ing K nowledge D istillation (RichKD), a novel cross-modal distillation framework that fuses the loits and features of a conventional teacher with those of a pre-trained CLIP model. Our key insight is that CLIP’s predictions, though not tailored to the target dataset , encapsulate complementary information in their broader semantic space. By fusing CLIP’s logits and features with those of the primary teacher, we effectively inject meaningful cross-modal perturbations into the supervision signal. This fusion acts as a soft ensemble of two diverse teachers; one grounded in dataset-specific discriminative patterns, and the other guided by semantic relationships learned from large-scale vision–language data. Figure 1 (a) illustrates the number of certain/uncertain and correct/incorrect predictions for the conventional teacher, CLIP, and their fusion. The fusion improves overall accuracy, increasing confident correct predictions by 3.6% and reducing confident mistakes by 7.7%. Figure 1 (b) presents two examples illustrating that perturbing the teacher’s logits with CLIP’s helps correct wrong predictions and refine the confidence distribution by better ranking non-target classes. To enrich the multimodal supervision, we further employ a multi-prompting strategy , generating multiple textual templates per class through CLIP’s text encoder and averaging the corresponding predictions. This multi-prompt ensemble captures different linguistic perspectives, smoothing the supervision and mitigating overfitting to specific textual formulations. The fused signals are then distilled into the student through a logit distillation loss. In summary, this paper makes the following contributions: • We investigate the integration of cross-modal knowledge into conventional knowledge distillation, an underexplored direction that leverages CLIP’s vision–language representations to enrich the teacher–student learning process. • We propose a simple yet effective logit–feature fusion mechanism that combines a dataset-specific teacher with CLIP’s multi-prompt predictions to construct semantically diverse and informative supervisory signals. • We provide both theoretical and empirical evidence that this cross-modal fusion enhances the diversity of teacher guidance, yielding improved performance and robustness compared to unimodal KD methods. 2 Related Work In this section, literature related to this work, including KD and multi-teacher KD, is presented. 2.1 Knowledge Distillation Knowledge distillation was popularized by KD 14 , who showed that a compact student can learn from a larger teacher by matching softened teacher logits. Following KD, FitNet 36 proposed to distill intermediate features, while RKD 34 focused on transferring relational knowledge to the student. Subsequent works can be categorized into three groups: logit-based 9 ; 56 ; 11 ; 30 ; 47 ; 6 ; 38 ; 22 ; 45 , feature-based 53 ; 13 ; 4 ; 10 ; 29 ; 25 , and similarity-based 44 ; 46 ; 24 ; 51 ; 16 ; 49 distillation methods. Furthermore, some studies have sought to enhance the distillation process using various techniques, such as adaptive distillation 58 ; 33 ; 3 ; 32 ; 17 ; 27 or modifying the teacher’s logits using label information 48 ; 1 ; 21 ; 42 . Although feature-based methods such as CRD 43 and SimKD 2 have achieved strong performance in KD, the dominant paradigm for classification remains logit distillation. Numerous logit-based methods have been proposed to improve the vanilla KD approach 56 ; 19 ; 23 ; 5 ; 41 . For example, DKD 56 decouples the KD loss into target and non-target class components; MLKD 19 introduces multi-level logit distillation; and CTKD 23 adopts a curriculum temperature strategy for logit distillation. More recently, NormKD 5 customizes the temperature for each sample, and LSKD 41 applies Z-score standardization to logits before the softmax operation. Figure 2 : Overall diagram of the proposed RichKD distillation method . CLIP’s logits and features are fused with those from the conventional teacher model. Feature and logit distillation losses are then defined between the fused representations and the student’s corresponding features and logits. During training phase, the parameters of CLIP and the teacher model are frozen, and the student is trained using feature and logit distillation losses in addition to the cross-entropy loss. Inconsistencies in feature dimensions are addressed through a linear layer transformation. 2.2 Multi-Teacher Distillation Using multiple teachers or auxiliary teacher assistants is a natural approach to provide richer supervision and to reduce the teacher–student capacity gap. Several strategies have been employed for this purpose, such as averaging logits 14 , using voting strategies 52 ; 55 , perturbing logits 39 ; 15 , or assigning teachers that specialize in different subsets of the dataset 18 . TAKD 31 was one of the early works that introduced the concept of a teacher

Read the complete text