Neural Attention Search
Difan Deng, Marius Lindauer · 2025
Read Neural Attention Search by Difan Deng, Marius Lindauer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13251.txt.
Description
Difan Deng and Marius Lindauer present a 2025 method to cut the memory cost of transformer models. When a transformer generates text, it stores a key and value for every past token. That store, the KV cache, grows fast and drives up running costs. Their framework, Neural Attention Search, learns which tokens still matter after a few steps. Tokens that no longer matter can be dropped from the cache.
The search sorts tokens into three types. Global tokens stay and can be read by all later tokens. Local tokens last until the next global token arrives. Sliding window tokens affect only a fixed span of followers. The type choices are learned together with the model's own weights through a trainable mask. Tests on new and fine tuned models showed large cache savings with performance held steady.
Catalog record
| CREATOR | Difan Deng, Marius Lindauer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13251