Long-Context Generalization with Sparse Attention
Pavlo Vasylenko, Marcos Treviso, André F. T. Martins · 2025
Read Long-Context Generalization with Sparse Attention by Pavlo Vasylenko, Marcos Treviso, André F. T. Martins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16640.txt.
Description
Pavlo Vasylenko and co-authors showed in a 2025 paper that sparse attention helps language models handle far longer texts. Standard attention spreads weight over every token, and as sequences grow, junk tokens soak up probability until the signal blurs. Sparse mechanisms based on entmax can assign exact zeros to irrelevant tokens. Their version adds a learnable temperature, sliding between sparse focus and dense spread as needed.
On synthetic tasks the method stretched context length up to a thousandfold beyond training. On language modeling it kept short-text skill while beating rivals at eight times the training length, with better perplexity trends and retrieval accuracy. The lesson: letting attention fully ignore tokens protects meaning when contexts get very long.
Catalog record
| CREATOR | Pavlo Vasylenko, Marcos Treviso, André F. T. Martins |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16640