ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads
Yifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren · 2025
Read ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads by Yifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03433.txt.
Description
Giant vision models know a lot, but bolting task-specific heads onto them is often clumsy and slow. Yifan Li and co-authors noticed the big models split naturally in two: early layers extract raw visual features, later layers adapt them. Their ViT-Split design freezes the whole model and hangs two light heads on it, one to learn the task, one to harvest the model's own multi-scale features as prior knowledge.
Dropping the usual extra convolutional branch removed a training bottleneck, and tuning only the heads cut overfitting. Training ran up to four times faster while matching or beating rival adapters on segmentation, detection, depth, and visual question answering. The lesson: respect the foundation model's internal division of labour instead of fighting it.
Catalog record
| CREATOR | Yifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03433