The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti · 2025
Read The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs by Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17768.txt.
Description
Piotr Nawrot and co-authors map the costs and gains of sparse attention in large language models in a 2025 paper. Attention is the step where a model weighs which past words matter, and it grows costly on long texts. Sparse versions skip most word pairs, but their trade-offs lacked a broad test.
The team ran the largest check of its kind, covering six methods, several model families, and texts up to 128,000 word pieces. They sort the methods by four design choices. Three lessons stand out: larger sparse models beat smaller full ones at equal cost, fine per-word picking during the first pass is not yet practical, and longer texts tolerate more skipping. The findings give builders clear rules for deployment.
Catalog record
| CREATOR | Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17768