Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

Dongwon Jo, Beomseok Kang, Jiwon Song, Jae-Joon Kim · 2026

Read Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection by Dongwon Jo, Beomseok Kang, Jiwon Song, Jae-Joon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03216.txt.

Description

Attention that drops and later reconsiders tokens is introduced by Dongwon Jo, together with 3 colleagues, for long-context inference. Attention's quadratic cost is the central bottleneck as contexts grow. Earlier methods use fixed sparse patterns or evict tokens permanently at certain layers. Token importance, though, varies by layer and head, so irreversible early decisions can discard information later layers would need for accurate answers.

Their Token Sparse Attention compresses queries, keys, and values per head to a reduced token set during attention, then decompresses the output back to full length. Tokens can thus be reconsidered in later layers. The method composes with existing dense and sparse attention implementations, including Flash Attention. Experiments showed a better accuracy-latency trade-off, with large attention speedups at very long contexts and minimal accuracy loss.

Catalog record

CREATORDongwon Jo, Beomseok Kang, Jiwon Song, Jae-Joon Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03216

Browse the full library · About the Tower of Babel