Vision Transformers Don’t Need Trained Registers
Nick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu · 2025
Read Vision Transformers Don’t Need Trained Registers by Nick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08010.txt.
Description
Nick Jiang and co-authors fixed vision transformer glitches without retraining in two thousand twenty-five. Some image tokens swell to huge norms in these models. The outliers muddy attention maps and hurt features. A known cure retrains models with register tokens. That cure costs a full training run. The team traced the glitch to sparse neurons.
Few neurons concentrate the runaway activations. Shifting their output into one extra token mimics registers. No weights change and no retraining happens. Attention and feature maps came out cleaner. Downstream scores improved over base models. Tests covered popular vision backbones. The fix applies to models already deployed.
Catalog record
| CREATOR | Nick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08010