Adaptive KV Cache Reuse for Fast Long-Context LLM Serving CCS: Computer systems organization Cloud computing CCS: Computing methodologies Natural language generation CCS: Information systems Information retrieval

Fei Li, Song Liu, Yan Liu, Jinhua Cui, Shiqiang Nie, Jinyu Wang, Weiguo Wu · 2026

Read Adaptive KV Cache Reuse for Fast Long-Context LLM Serving CCS: Computer systems organization Cloud computing CCS: Computing methodologies Natural language generation CCS: Information systems Information retrieval by Fei Li, Song Liu, Yan Liu, Jinhua Cui, Shiqiang Nie, Jinyu Wang, Weiguo Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24022.txt.

Description

Li and six co-authors speed long-context serving with CacheTune, a system that reuses key-value cache beyond strict prefixes. It spots, through frequency analysis, the cache pairs most vital to cross-attention recovery, recomputing only those tokens online while reusing the rest. Sparse transfers, multi-stream overlap, deferred positional recovery, and hardware-aware tuning turn that selection into end-to-end latency cuts. Tests show first-token speedups of 3.72 to 4.86 times and throughput gains of 3.93 to 6.21 times, with quality near full recompute.

Catalog record

CREATORFei Li, Song Liu, Yan Liu, Jinhua Cui, Shiqiang Nie, Jinyu Wang, Weiguo Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54795
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24022

Browse the full library · About the Tower of Babel