Simplifying Knowledge Transfer in Pretrained Models
Siddharth Jain [email protected], Shyamgopal Karthik [email protected], Vineet Gandhi [email protected] · 2025
Read Simplifying Knowledge Transfer in Pretrained Models by Siddharth Jain [email protected], Shyamgopal Karthik [email protected], Vineet Gandhi [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22208.txt.
Introduction
Siddharth Jain, Shyamgopal Karthik, and Vineet Gandhi let pretrained models teach each other. Different architectures, trained differently, absorb different lessons from the same data, so one model's blind spot is often another's strength. The authors treat public model repositories as a standing faculty rather than a shelf of finished products.
Their scheme partitions the data so each pretrained model settles into the role it suits, student or teacher, on its own. Knowledge then flows both ways across the pairing. In image classification, two vision Transformers of different sizes improved each other, with the larger gaining over a percentage point. In semantic segmentation every metric rose with transfer both within and across backbone families; and in video saliency prediction the same exchange helped again. The method needs no new pretraining, only a structured conversation between models that already exist.
Catalog record
| CREATOR | Siddharth Jain [email protected], Shyamgopal Karthik [email protected], Vineet Gandhi [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22208
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Simplifying Knowledge Transfer in Pretrained Models Siddharth Jain [email protected], Shyamgopal Karthik [email protected], Vineet Gandhi [email protected] Abstract Pretrained models are ubiquitous in the current deep learning landscape, offering strong results on a broad range of tasks. Recent works have shown that models differing in various design choices exhibit categorically diverse generalization behavior, resulting in one model grasping distinct data-specific insights unavailable to the other. In this paper, we propose to leverage large publicly available model repositories as an auxiliary source of model improvements. We introduce a data partitioning strategy where pretrained models autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge. Experiments across various tasks demonstrate the effectiveness of our proposed approach. In image classification, we improved the performance of ViT-B by approximately 1.4% through bidirectional knowledge transfer with ViT-T. For semantic segmentation, our method boosted all evaluation metrics by enabling knowledge transfer both within and across backbone architectures. In video saliency prediction, our approach achieved a new state-of-the-art. We further extend our approach to knowledge transfer between multiple models, leading to considerable performance improvements for all model participants. The code is available at: https://github.com/Syd-J/Bi-KD 1 Introduction Knowledge Distillation (KD) ( Buciluǎ et al. 2006 ; Hinton et al. 2015 ; Beyer et al. 2022 ) intends to transfer knowledge from a large ‘teacher’ model to a smaller ‘student’ model. Traditional KD methods utilize the predictions from the pretrained teacher model to supervise the training of the student model, encouraging it to generalize better than if it were trained from scratch alone. However, vanilla KD is a two-stage process that begins with training a teacher model and then freezing it to distill knowledge into the student model, meaning that the knowledge can only be transferred from the teacher to the student. Online Knowledge Distillation methods ( Zhang et al. 2018 ; Guo et al. 2020 ) overcome this limitation by adopting a one-stage training process, jointly training a set of student models that learn from each other in a peer-teaching manner. Although online KD methods employ a single stage training process, they distill knowledge into untrained student models. Transferring knowledge in this way neglects the existence of complementary knowledge between pretrained models, a factor that, if considered, can enhance generalization ( Gontijo-Lopes et al. 2022 ; Roth et al. 2024 ) . Various design choices such as hyperparameters, model architecture, optimization strategies, and pretraining dataset shape the semantic knowledge a model acquires ( Bouthillier et al. 2021 ; Wagner et al. 2022 ) . Gontijo-Lopes et al. 2022 show that model pairs with diverging training methodologies produce increasingly uncorrelated errors. Even low-accuracy models may capture some data-specific insights that high-accuracy models might overlook. Roth et al. 2024 capitalize on this finding by transferring complementary knowledge between any pretrained teacher and student model pair. They propose a data partitioning strategy that divides the dataset into instances where knowledge transfer from a teacher is beneficial and those where retaining the student’s behavior is preferred. However, their approach operates in a unidirectional manner, wherein the teacher remains fixed after pretraining and the student is trained to minimize the teacher-student gap. This contrasts with the real-world teacher-student dynamic, in which a teacher continuously improves their knowledge and teaching skills through ongoing interactions with the student ( Cornelius-White 2007 ; Wright 2011 ) . Figure 1 : The figure illustrates the training process for a given batch of samples. (a) KD ( Hinton et al. 2015 ) utilizes a pretrained teacher model and trains a student model to mimic the teacher’s predictions on each sample. (b) KL+DP ( Roth et al. 2024 ) employs a frozen teacher and the original frozen student (called student-teacher) to jointly guide the training of the student model. They partition the dataset into samples where learning from the teacher is desired and ones where knowledge from the original frozen student should be retained. (c) In Bi-KD (ours), both models are trainable and learn from each other on every sample, resulting in bidirectional knowledge transfer. Different from KD and KL+DP, we are able to improve both models simultaneously. Transferring knowledge among a group of pretrained models lets each one benefit from the unique strengths and insights that the others have already developed. They can complement each other’s weaknesses and build more robust, generalized representations. When models are trained together, they iteratively update their predictions, effectively “teaching” each other. This simultaneous updating can lead to a performance boost that none of the models might achieve if trained independently ( Zhang et al. 2018 ; Zhu et al. 2018 ; Guo et al. 2020 ) . To this end, we propose a simple method for bidirectional knowledge transfer between pretrained models, a challenging task due to their prior training. Building on the data partition strategy introduced by Roth et al. 2024 , we dynamically assign the teacher role to the model with the highest prediction confidence for the ground-truth class, allowing it to transfer knowledge to the other model. Unlike their fixed partitioning approach, our confidence-based data partitioning evolves as the models improve, adapting throughout the training process (as illustrated in Figure 1 ). Despite the apparent simplicity of our method, it leads to consistent performance improvements for both the models within a single training stage. We validated our approach through extensive experiments on models with diverse architectures, performance levels, sizes, and training objectives. Our method was tested across multiple tasks, including image classification, semantic segmentation, and video saliency prediction, and was further extended to enable parallel knowledge transfer among multiple models. In all cases, we observed consistent performance improvements. Overall, we make the following contributions: • We demonstrate the ability for bidirectional knowledge transfer in pretrained models. Specifically, we show that knowledge can be transferred across both models simultaneously. • We provide experiments across ImageNet classification, semantic segmentation, and video saliency prediction, where we observe consistent improvements for all participating models. In particular, our method sets a new state-of-the-art in video saliency prediction. • We establish that our framework seamlessly extends to concurrent knowledge transfer across multiple models, thereby progressively enhancing the performance of each individual model as additional models are integrated. 2 Related Work Knowledge Distillation (KD), pioneered by Buciluǎ et al. 2006 , aimed to compress large teacher models into smaller student models by aligning their soft target distributions. Hinton et al. 2015 refined this approach by incorporating temperature scaling to minimize the difference between the softened class probabilities of the teacher and student models. Beyer et al. 2022 further highlighted the importance of consistent image augmentations and extended training schedules for effective KD. Building on these ideas, recent works have explored transferring knowledge beyond just output probabilities. Romero et al. 2014 proposed aligning the intermediate feature representations between the teacher and the student models, while Zagoruyko & Komodakis 2017 train a student to imitate the attention maps of teacher networks. Park et al. 2019 introduced a method that preserves the structural relationship between the outputs using distance-wise and angle-wise losses. However, these methods often require careful layer selection and loss balancing ( Yun et al. 2019 ) , making them highly dependent on specific network architectures. To address this, Srinivas & Fleuret