Estimating the Effective Rank of Vision Transformers via Low-Rank Factorization

Liyu Zerihun · 2025

Read Estimating the Effective Rank of Vision Transformers via Low-Rank Factorization by Liyu Zerihun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00792.txt.

Description

Liyu Zerihun wrote a 2025 paper on how much capacity vision transformers really use. These models are heavily over-parameterized, meaning they carry far more weights than the task may need. Yet the patterns they learn often sit in a much smaller, low-rank space.

Zerihun's framework estimates a model's intrinsic dimensionality, its effective rank. First, a full-size teacher model is trained. Then its weights are factorized at several ranks, and smaller student models learn from the teacher by distillation. Watching how accuracy changes with rank shows where the real capacity lies. That knowledge can guide leaner designs.

Catalog record

CREATORLiyu Zerihun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00792

Browse the full library · About the Tower of Babel