Linear-Time Global Visual Modeling without Explicit Attention

Ruize He Dongchen Han Gao Huang Tsinghua University · 2026

Read Linear-Time Global Visual Modeling without Explicit Attention by Ruize He Dongchen Han Gao Huang Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01711.txt.

Description

Ruize He, Dongchen Han, and Gao Huang argue that attention, the Transformer's signature move, is secretly a different machine than it looks. Reframed mathematically, attention acts as a small network whose weights are predicted on the fly from the input. Its global reach comes from those dynamic weights holding a compressed summary of the whole sequence.

If that reading is right, explicit token-to-token attention, with its square-law cost, may be replaceable. The team design several ways to predict parameters dynamically and slot them into ordinary network layers, keeping cost linear in sequence length. Vision experiments across models show the dynamic-parameter route matching the global modeling power that attention was credited with. The paper offers both a lens on why attention works and a candidate engine for cheaper sequence models.

Catalog record

CREATORRuize He Dongchen Han Gao Huang Tsinghua University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01711

Browse the full library · About the Tower of Babel