KV Cache Offloading for Context-Intensive Tasks

Andrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU · 2026

Read KV Cache Offloading for Context-Intensive Tasks by Andrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08426.txt.

Description

Andrey Bocharnikov and four colleagues test a popular memory saving trick for long context language models on tasks that demand heavy lookup from the prompt. Key value cache offloading, they find, degrades sharply on such tasks for both Llama 3 and Qwen 3. Their new Text2JSON benchmark exposes the failure, which they trace to weak key projections and unreliable landmarks, and a simpler rival strategy restores accuracy across model families.

Catalog record

CREATORAndrey BocharnikovHSE, Yandex, Ivan ErmakovHSE, Yandex, Denis KuznedelevYandex, Vyacheslav ZhdanovskiyYandex, Yegor YershovYandex, NSU
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29310
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08426

Browse the full library · About the Tower of Babel