ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads

Yifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren · 2025

Read ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads by Yifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03433.txt.

Description

Giant vision models know a lot, but bolting task-specific heads onto them is often clumsy and slow. Yifan Li and co-authors noticed the big models split naturally in two: early layers extract raw visual features, later layers adapt them. Their ViT-Split design freezes the whole model and hangs two light heads on it, one to learn the task, one to harvest the model's own multi-scale features as prior knowledge.

Dropping the usual extra convolutional branch removed a training bottleneck, and tuning only the heads cut overfitting. Training ran up to four times faster while matching or beating rival adapters on segmentation, detection, depth, and visual question answering. The lesson: respect the foundation model's internal division of labour instead of fighting it.

Catalog record

CREATORYifan Li, Xin Li, Tianqin Li, Wenbin He, Yu Kong, Liu Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03433

Browse the full library · About the Tower of Babel