Vision Transformers Don’t Need Trained Registers

Nick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu · 2025

Read Vision Transformers Don’t Need Trained Registers by Nick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08010.txt.

Description

Nick Jiang and co-authors fixed vision transformer glitches without retraining in two thousand twenty-five. Some image tokens swell to huge norms in these models. The outliers muddy attention maps and hurt features. A known cure retrains models with register tokens. That cure costs a full training run. The team traced the glitch to sparse neurons.

Few neurons concentrate the runaway activations. Shifting their output into one extra token mimics registers. No weights change and no retraining happens. Attention and feature maps came out cleaner. Downstream scores improved over base models. Tests covered popular vision backbones. The fix applies to models already deployed.

Catalog record

CREATORNick Jiang, Amil DravidAlexei A. EfrosYossi Gandelsman UC Berkeley, Equal contribution {nickj,amil_dravid,aaefros,yossi_gandelsman}@berkeley.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08010

Browse the full library · About the Tower of Babel