The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti · 2025

Read The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs by Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17768.txt.

Description

Piotr Nawrot and co-authors map the costs and gains of sparse attention in large language models in a 2025 paper. Attention is the step where a model weighs which past words matter, and it grows costly on long texts. Sparse versions skip most word pairs, but their trade-offs lacked a broad test.

The team ran the largest check of its kind, covering six methods, several model families, and texts up to 128,000 word pieces. They sort the methods by four design choices. Three lessons stand out: larger sparse models beat smaller full ones at equal cost, fine per-word picking during the first pass is not yet practical, and longer texts tolerate more skipping. The findings give builders clear rules for deployment.

Catalog record

CREATORPiotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17768

Browse the full library · About the Tower of Babel