KV Cache Offloading for Context-Intensive Tasks
Andrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU · 2026
Read KV Cache Offloading for Context-Intensive Tasks by Andrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08426.txt.
Description
Andrey Bocharnikov and four colleagues test a popular memory saving trick for long context language models on tasks that demand heavy lookup from the prompt. Key value cache offloading, they find, degrades sharply on such tasks for both Llama 3 and Qwen 3. Their new Text2JSON benchmark exposes the failure, which they trace to weak key projections and unreliable landmarks, and a simpler rival strategy restores accuracy across model families.
Catalog record
| CREATOR | Andrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29310 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08426