Feature-based One-For-All: Heterogeneous Distillation Across Vision Architectures

Jhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng · 2025

Read Feature-based One-For-All: Heterogeneous Distillation Across Vision Architectures by Jhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08885.txt.

Description

Jhe-Hao Lin and five co-workers wrote a 2025 paper on teaching small vision models with large ones across distinct designs. Knowledge distillation copies a heavy teacher's skill into a light student. Past methods assumed teacher and student share a design family. New designs, from classic networks to transformers, break that assumption.

Their frame adds prompt blocks that take student feedback, so teacher features adapt as the student learns. A region-aware attention step fixes view mismatch between designs. Tests on three image data sets beat prior methods. The authors share code for the frame. Mixed-design teaching is the core advance claimed.

Catalog record

CREATORJhe-Hao Lin, Yi Yao, Chang-Feng Hu, Hongxia Xie, Hong-Han Shuai, Wen-Huang Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-08885

Browse the full library · About the Tower of Babel