Linear-Time Global Visual Modeling without Explicit Attention
Ruize He Dongchen Han Gao Huang Tsinghua University · 2026
Read Linear-Time Global Visual Modeling without Explicit Attention by Ruize He Dongchen Han Gao Huang Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01711.txt.
Description
Ruize He, Dongchen Han, and Gao Huang argue that attention, the Transformer's signature move, is secretly a different machine than it looks. Reframed mathematically, attention acts as a small network whose weights are predicted on the fly from the input. Its global reach comes from those dynamic weights holding a compressed summary of the whole sequence.
If that reading is right, explicit token-to-token attention, with its square-law cost, may be replaceable. The team design several ways to predict parameters dynamically and slot them into ordinary network layers, keeping cost linear in sequence length. Vision experiments across models show the dynamic-parameter route matching the global modeling power that attention was credited with. The paper offers both a lens on why attention works and a candidate engine for cheaper sequence models.
Catalog record
| CREATOR | Ruize He Dongchen Han Gao Huang Tsinghua University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01711