Estimating the Effective Rank of Vision Transformers via Low-Rank Factorization
Liyu Zerihun · 2025
Read Estimating the Effective Rank of Vision Transformers via Low-Rank Factorization by Liyu Zerihun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00792.txt.
Description
Liyu Zerihun wrote a 2025 paper on how much capacity vision transformers really use. These models are heavily over-parameterized, meaning they carry far more weights than the task may need. Yet the patterns they learn often sit in a much smaller, low-rank space.
Zerihun's framework estimates a model's intrinsic dimensionality, its effective rank. First, a full-size teacher model is trained. Then its weights are factorized at several ranks, and smaller student models learn from the teacher by distillation. Watching how accuracy changes with rank shows where the real capacity lies. That knowledge can guide leaner designs.
Catalog record
| CREATOR | Liyu Zerihun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00792