FASA: Frequency-Aware Sparse Attention
Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley · 2026
Read FASA: Frequency-Aware Sparse Attention by Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03152.txt.
Description
Long inputs exhaust language model memory, and Yifei Wang heads a team of 8 other researchers with a frequency-based fix. The key-value cache grows with input length and bottlenecks deployment. Token pruning keeps only a critical subset, but static choices risk irreversible loss while dynamic heuristics miss how importance depends on the query being answered for each new request.
Their framework, FASA, predicts token importance in a query-aware way. The key insight concerns rotary position embeddings, where a few dominant frequency chunks agree strongly with full attention behavior. Those chunks identify critical tokens cheaply, and focused attention runs only on them. Across long-context tasks, FASA beat token-eviction baselines and neared oracle accuracy. It kept near-full performance while retaining only a small cache fraction.
Catalog record
| CREATOR | Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03152