Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei · 2026
Read Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM by Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29563.txt.
Description
Roy, Zhai, Hajimirsadeghi, and Samei attack the memory cost of running large language models on long inputs. During inference, models store a key-value cache, the remembered keys and values for earlier tokens. That cache grows heavy, alongside the quadratic cost of self-attention. Eviction strategies drop less important tokens, but they often cause a steep performance drop on long-context reasoning.
The team traces that drop to reduced coverage: evicted caches cover fewer unique tokens. They also show in theory that lower coverage caps the information shared between inputs and outputs. Their method, K-VEC, makes eviction coverage-aware. Cross-head and cross-layer coverage modules protect token diversity across attention heads and model layers. On sixteen LongBench subsets, K-VEC improves scores by up to about ten points over existing methods at the same eviction rate and memory budget.
Catalog record
| CREATOR | Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29563