The Linear Attention Resurrection in Vision Transformer
Chuanyang Zheng · 2025
Read The Linear Attention Resurrection in Vision Transformer by Chuanyang Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16182.txt.
Description
Chuanyang Zheng, a solo author, presents a faster attention method for vision transformers in a 2025 paper. Vision transformers read images with an attention step whose cost grows very fast as images get larger, which blocks use on high detail images. Prior fast methods often lost the model's ability to take in the whole image at once.
Zheng shows that fast linear attention lacks a key trait of the standard method: it does not focus its weights sharply. A new local focus module restores that trait. The full model keeps global reach at linear cost and scored well on image labeling tests. It also worked well as a base for object finding and image segmentation tasks.
Catalog record
| CREATOR | Chuanyang Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16182