Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei · 2026

Read Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM by Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29563.txt.

Description

Roy, Zhai, Hajimirsadeghi, and Samei attack the memory cost of running large language models on long inputs. During inference, models store a key-value cache, the remembered keys and values for earlier tokens. That cache grows heavy, alongside the quadratic cost of self-attention. Eviction strategies drop less important tokens, but they often cause a steep performance drop on long-context reasoning.

The team traces that drop to reduced coverage: evicted caches cover fewer unique tokens. They also show in theory that lower coverage caps the information shared between inputs and outputs. Their method, K-VEC, makes eviction coverage-aware. Cross-head and cross-layer coverage modules protect token diversity across attention heads and model layers. On sixteen LongBench subsets, K-VEC improves scores by up to about ten points over existing methods at the same eviction rate and memory budget.

Catalog record

CREATORShuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29563

Browse the full library · About the Tower of Babel