FASA: Frequency-Aware Sparse Attention

Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley · 2026

Read FASA: Frequency-Aware Sparse Attention by Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03152.txt.

Description

Long inputs exhaust language model memory, and Yifei Wang heads a team of 8 other researchers with a frequency-based fix. The key-value cache grows with input length and bottlenecks deployment. Token pruning keeps only a critical subset, but static choices risk irreversible loss while dynamic heuristics miss how importance depends on the query being answered for each new request.

Their framework, FASA, predicts token importance in a query-aware way. The key insight concerns rotary position embeddings, where a few dominant frequency chunks agree strongly with full attention behavior. Those chunks identify critical tokens cheaply, and focused attention runs only on them. Across long-context tasks, FASA beat token-eviction baselines and neared oracle accuracy. It kept near-full performance while retaining only a small cache fraction.

Catalog record

CREATORYifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03152

Browse the full library · About the Tower of Babel