Block-Recurrent Dynamics in ViTs

Mozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor · 2025

Read Block-Recurrent Dynamics in ViTs by Mozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19941.txt.

Description

Mozes Jacobs and five colleagues argue that vision transformers secretly run a much smaller program than their depth suggests. They call the claim the Block-Recurrent Hypothesis: a trained transformer's many blocks can be rewritten as a handful of distinct blocks applied over and over. Similarity patterns between layers hint at a few contiguous computational phases, but resemblance alone does not prove the computation is reusable.

So they trained Raptor, block-recurrent surrogates of pretrained vision transformers, and fit them well; one recovered 96 percent of a DINOv2 model's linear-probe accuracy using only two distinct blocks at equal runtime. Reading the recurrent dynamics then revealed trajectories settling into class-dependent directions, token-specific behaviors, and late layers collapsing toward low-rank updates. Depth, on this view, iterates a compact loop rather than executing dozens of unique steps.

Catalog record

CREATORMozes Jacobs, Thomas Fel, Richard HakimAlessandra BrondettaDemba BaT. Andy Keller Kempner Institute, Harvard UniversityUniversity of OsnabrückHarvard University kempnerinstitute.github.io/raptor
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19941

Browse the full library · About the Tower of Babel