Block-Recurrent Dynamics in ViTs
Mozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor · 2025
Read Block-Recurrent Dynamics in ViTs by Mozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19941.txt.
Description
Mozes Jacobs and five colleagues argue that vision transformers secretly run a much smaller program than their depth suggests. They call the claim the Block-Recurrent Hypothesis: a trained transformer's many blocks can be rewritten as a handful of distinct blocks applied over and over. Similarity patterns between layers hint at a few contiguous computational phases, but resemblance alone does not prove the computation is reusable.
So they trained Raptor, block-recurrent surrogates of pretrained vision transformers, and fit them well; one recovered 96 percent of a DINOv2 model's linear-probe accuracy using only two distinct blocks at equal runtime. Reading the recurrent dynamics then revealed trajectories settling into class-dependent directions, token-specific behaviors, and late layers collapsing toward low-rank updates. Depth, on this view, iterates a compact loop rather than executing dozens of unique steps.
Catalog record
| CREATOR | Mozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19941