From independent patches to coordinated attention: Controlling information flow in vision transformers
Kieran A. Murphy · 2026
Read From independent patches to coordinated attention: Controlling information flow in vision transformers by Kieran A. Murphy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04784.txt.
Description
Kieran A. Murphy makes attention in vision transformers pay for the information it carries, in a 2026 study of how image patches communicate. Normally each patch can attend globally with no explicit cost, which makes internal information flow hard to measure or control.
Murphy inserts variational information bottlenecks, adjustable constrictions, on every attention-mediated write into the model's shared stream. Training with an explicit information cost yields a spectrum of models. At one end, patches process independently. At the other, attention is fully expressive. On an image classification benchmark, he charts how behavior and routing evolve across that spectrum. Early attention heads that first transmit information reveal how global representations emerge from local processing.
Catalog record
| CREATOR | Kieran A. Murphy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04784