Conf-KV : Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM Inference

Yubo Li Yidi Miao · 2026

Read Conf-KV : Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM Inference by Yubo Li Yidi Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24786.txt.

Description

Yubo Li and Yidi Miao present CONF-KV, a cache manager that sets the key-value budget at each step from model uncertainty. Long chats make the cache the main memory cost, yet most eviction rules use fixed windows or past attention. CONF-KV turns the next-token distribution into a confidence score. It keeps more context when unsure and prunes hard when sure. Paired with mixed-precision storage, it reaches 91 percent retrieval accuracy on a needle-in-a-haystack test.

Catalog record

CREATORYubo Li Yidi Miao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52474
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24786

Browse the full library · About the Tower of Babel