Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention

Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, Jian Tang · 2025

Read Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention by Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, Jian Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00449.txt.

Description

Zhihao Zhan and co-authors analyze in 2025 why state-space models falter on very long texts. These models promise fast handling of long sequences, unlike the dominant attention design whose cost grows steeply. Yet they struggle to recall distant facts, and the usual test task, recalling one key without context, is too simple to show why.

The team defines a harder joint recall task where the right value depends on context. They prove state-space models cannot solve it in fast time, then blend them with sparse attention that can. A hashing-based version tuned for language beat state-space baselines on both made-up and real long-text benchmarks.

Catalog record

CREATORZhihao Zhan, Jianan Zhao, Zhaocheng Zhu, Jian Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00449

Browse the full library · About the Tower of Babel