Feature-based One-For-All: Heterogeneous Distillation Across Vision Architectures
Jhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng · 2025
Read Feature-based One-For-All: Heterogeneous Distillation Across Vision Architectures by Jhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08885.txt.
Description
Jhe-Hao Lin and five co-workers wrote a 2025 paper on teaching small vision models with large ones across distinct designs. Knowledge distillation copies a heavy teacher's skill into a light student. Past methods assumed teacher and student share a design family. New designs, from classic networks to transformers, break that assumption.
Their frame adds prompt blocks that take student feedback, so teacher features adapt as the student learns. A region-aware attention step fixes view mismatch between designs. Tests on three image data sets beat prior methods. The authors share code for the frame. Mixed-design teaching is the core advance claimed.
Catalog record
| CREATOR | Jhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-08885