RAttention : Towards the Minimal Sliding Window Size in Local-Global Attention Models

Bailin Wang Chang Lan Chong Wang Ruoming Pang · 2025

Read RAttention : Towards the Minimal Sliding Window Size in Local-Global Attention Models by Bailin Wang Chang Lan Chong Wang Ruoming Pang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15545.txt.

Description

Wang and co-authors shrink the attention window of language models without losing quality, in a 2025 machine learning paper. Local-global attention models read nearby tokens closely and skim the rest, but window size is a hard trade-off. Big windows keep quality yet save little, while small windows degrade answers because outside tokens vanish entirely. Their variant, RAttention, pairs local attention with a linear mechanism that keeps a running trace of distant tokens.

Pretraining at two scales showed a window of just 512 tokens matching full attention across settings. The recurrent trace also helped on long-context tests. A custom kernel kept training speed intact. The authors argue the design shifts the efficiency frontier for short and long inputs alike, which matters as models serve ever longer documents.

Catalog record

CREATORBailin Wang Chang Lan Chong Wang Ruoming Pang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15545

Browse the full library · About the Tower of Babel