CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation
Jungsoo Lee Debasmit Das Munawar Hayat, Sungha Choi Kyuwoong Hwang Fatih Porikli, Qualcomm AI Research {jungsool, debadas, hayat, sunghac, kyuwoong, fporikli}@qti.qualcomm.com · 2025
Read CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation by Jungsoo Lee Debasmit Das Munawar Hayat, Sungha Choi Kyuwoong Hwang Fatih Porikli, Qualcomm AI Research {jungsool, debadas, hayat, sunghac, kyuwoong, fporikli}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18244.txt.
Introduction
Jungsoo Lee and co-authors present CustomKD, a way to distill giant vision models into small edge models without the usual mismatch losses, in a 2025 paper. Foundation vision models see the world richly but cannot run on phones. Small models like MobileNet need that knowledge transferred. A huge capacity gap and unlike architectures block naive distillation.
Bigger teachers, the team notes, do not automatically teach better. Their method customizes what the foundation model passes down. The student improves on downstream tasks despite the gap. Qualcomm's labs target real on-device deployment. Edge AI gains a recipe for smarter small models. The work also questions bigger-is-better teacher folklore. Practical vision products are the end audience.
Catalog record
| CREATOR | Jungsoo Lee Debasmit Das Munawar Hayat, Sungha Choi Kyuwoong Hwang Fatih Porikli, Qualcomm AI Research {jungsool, debadas, hayat, sunghac, kyuwoong, fporikli}@qti.qualcomm.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation Jungsoo Lee Debasmit Das Munawar Hayat, Sungha Choi Kyuwoong Hwang Fatih Porikli, Qualcomm AI Research {jungsool, debadas, hayat, sunghac, kyuwoong, fporikli}@qti.qualcomm.com Abstract We propose a novel knowledge distillation approach, CustomKD, that effectively leverages large vision foundation models (LVFMs) to enhance the performance of edge models ( e.g . , MobileNetV3). Despite recent advancements in LVFMs, such as DINOv2 and CLIP, their potential in knowledge distillation for enhancing edge models remains underexplored. While knowledge distillation is a promising approach for improving the performance of edge models, the discrepancy in model capacities and heterogeneous architectures between LVFMs and edge models poses a significant challenge. Our observation indicates that although utilizing larger backbones ( e.g . , ViT-S to ViT-L) in teacher models improves their downstream task performances, the knowledge distillation from the large teacher models fails to bring as much performance gain for student models as for teacher models due to the large model discrepancy. Our simple yet effective CustomKD customizes the well-generalized features inherent in LVFMs to a given student model in order to reduce model discrepancies. Specifically, beyond providing well-generalized original knowledge from teachers, CustomKD aligns the features of teachers to those of students, making it easy for students to understand and overcome the large model discrepancy overall. CustomKD significantly improves the performances of edge models in scenarios with unlabeled data such as unsupervised domain adaptation ( e.g . , OfficeHome and DomainNet) and semi-supervised learning ( e.g . , CIFAR-100 and ImageNet), achieving the new state-of-the-art performances. † † † Corresponding author. ‡ Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction Recent efforts in computer vision have focused on building large vision foundation models (LVFMs), with a substantial number of parameters, generally trained using large-scale pretaining datasets. Owing to their well-generalized representation, LVFMs are widely known to achieve state-of-the-art performances on diverse downstream tasks Oquab et al. 2023 ; Radford et al. 2021 ; Ilharco et al. 2021 ; Kirillov et al. 2023 ; Liu et al. 2022a . However, while LVFMs have demonstrated impressive performance, their utilization in resource-constrained real-world applications is challenging due to their high computational costs and extensive parameters, which often impede their deployment Firoozi et al. 2023 ; Yan et al. 2024 ; Wang et al. 2023a . Consequently, edge models, known for their computational efficiency, may remain the preferred choice for real-world applications, especially when considering deployment on mobile devices. However, these edge models typically offer limited performance, so finding a solution to improve their capabilities while maintaining their low computation costs is necessary. While training edge models with more labeled data samples to improve their performance on a certain downstream task would be one viable solution, collecting more labeled samples is labor-intensive and expensive. Without further data collection, one feasible solution to tackle such a challenge is performing knowledge distillation (KD) Peng et al. 2019 ; Park et al. 2019 ; Ba and Caruana 2014 ; Zhao et al. 2022 ; Ji et al. 2021 with LVFMs serving as teachers for the edge models. This approach can enhance the performance of edge models to be comparable to that of LVFMs while maintaining their low computational costs. Specifically, by leveraging unlabeled data samples, we can extract meaningful information from LVFMs and transfer it to edge models without incurring expensive data annotation costs. However, one major challenge of using LVFMs as teachers is the model discrepancy between LVFMs and edge models, leading to differences in representation spaces. In this KD setting, the model discrepancy arises from two primary factors. First, LVFMs are built with a vast number of parameters in order to understand the massive amount of knowledge from large-scale pretraining datasets while edge models are built on only a limited number of parameters. Second, the heterogeneous architectures ( i.e . , architectural difference) between students ( e.g . , CNN based) and teachers ( e.g . , ViT based) may also cause the model discrepancy. To be more specific, prevalent LVFMs employ ViT-based architectures Oquab et al. 2023 ; Ilharco et al. 2021 ; Liu et al. 2021 ; Liu et al. 2022a ; Yuan et al. 2021 while CNN-based edge models are favored Guo 2017 ; Bolhasani et al. 2021 ; Yu et al. 2022 for their fast inference speed. Our preliminary experiments demonstrate that the existing KD methods fail to effectively improve the performance of student model when the model discrepancy is substantial (Section 3.2 ). Specifically, when utilizing larger backbones of teacher models ( e.g . , changing the backbone from ViT-S to ViT-L), we observe that existing KD methods fail to further improve the performances of edge models as much as the improved performances of teachers. To address this issue, we propose a simple yet effective feature alignment method called CustomKD, which alternates between two stages: 1) feature customization and 2) knowledge distillation. In the feature customization stage, we customize the well-generalized features of LVFMs to a given student model by aligning them to the representation space of the student using the student’s head classifier. In the KD stage, we encourage the student model to imitate two different features: 1) the task-general feature extracted from the frozen teacher, and 2) the customized task-specific feature obtained during the feature customization stage. Since the student model, including its head classifier, is updated during the KD stage, we alternate these two stages to progressively enhance the student model by continuously improving task-specific features. Importantly, we do not alter the architectures or inference processes of edge models, allowing us to significantly improve their performance without increasing inference speed. Although feature alignment has been widely adopted and explored in various studies Sun and Saenko 2016 ; Jung et al. 2023 ; Zhu and Wang 2021 ; Ben-Baruch et al. 2022 ; Romero et al. 2015 , we want to emphasize that finding how to perform feature alignment using LVFMs as teachers in KD is underexplored and our technical novelty lies in finding answer to such a challenge. The major contributions of this paper are: \bullet Although using larger backbones improves the downstream task performances of LVFMs, our experiment demonstrates that existing KD methods fail to bring as much performance gain for edge models as for LVFMs. \bullet We propose CustomKD, a knowledge distillation method that enables to leverage LVFMs with large backbones as teachers by overcoming the large model discrepancy. \bullet CustomKD consistently improves the performances of edge models on various tasks ( e.g . , unsupervised domain adaptation and semi-supervised learning), without architectural changes nor increased inference speed. 2 Related Work 2.1 Large Vision Foundation Models The unprecedented breakthroughs of large language models in natural language processing have motivated computer vision studies to build large models for computer vision tasks. Due to such efforts, large vision foundation models (LVFMs) have received attention for their outstanding performances in diverse downstream tasks Oquab et al. 2023 ; Radford et al. 2021 ; Ilharco et al. 2021 ; Kirillov et al. 2023 ; Liu et al. 2022a ; Yuan et al. 2021 , even with a simple linear probing. In this work, the definition of LVFMs includes the image encoders