The Linear Attention Resurrection in Vision Transformer

Chuanyang Zheng · 2025

Read The Linear Attention Resurrection in Vision Transformer by Chuanyang Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16182.txt.

Description

Chuanyang Zheng, a solo author, presents a faster attention method for vision transformers in a 2025 paper. Vision transformers read images with an attention step whose cost grows very fast as images get larger, which blocks use on high detail images. Prior fast methods often lost the model's ability to take in the whole image at once.

Zheng shows that fast linear attention lacks a key trait of the standard method: it does not focus its weights sharply. A new local focus module restores that trait. The full model keeps global reach at linear cost and scored well on image labeling tests. It also worked well as a base for object finding and image segmentation tasks.

Catalog record

CREATORChuanyang Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16182

Browse the full library · About the Tower of Babel