Neural Attention Search

Difan Deng, Marius Lindauer · 2025

Read Neural Attention Search by Difan Deng, Marius Lindauer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13251.txt.

Description

Difan Deng and Marius Lindauer present a 2025 method to cut the memory cost of transformer models. When a transformer generates text, it stores a key and value for every past token. That store, the KV cache, grows fast and drives up running costs. Their framework, Neural Attention Search, learns which tokens still matter after a few steps. Tokens that no longer matter can be dropped from the cache.

The search sorts tokens into three types. Global tokens stay and can be read by all later tokens. Local tokens last until the next global token arrives. Sliding window tokens affect only a fixed span of followers. The type choices are learned together with the model's own weights through a trainable mask. Tests on new and fine tuned models showed large cache savings with performance held steady.

Catalog record

CREATORDifan Deng, Marius Lindauer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13251

Browse the full library · About the Tower of Babel