KeyDiff : Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, Chris Lott · 2025
Read KeyDiff : Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments by Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, Chris Lott (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15364.txt.
Description
Junyoung Park and colleagues present KeyDiff, a method to trim the key-value cache in long-context language models. Their 2025 arXiv paper starts from a geometric observation. Keys that stand out in shape tend to draw high attention scores. A cache must drop tokens when memory is tight. The task is to keep the tokens that matter most.
KeyDiff ranks tokens by key similarity alone, with no training needed. Diverse keys are kept, since they likely carry high attention. The method never needs raw attention scores to decide. That lets it pair with fast attention tools like FlashAttention. Tests on Llama and Qwen models stayed near baseline quality. Cache size fell by about a quarter under strict budgets.
Catalog record
| CREATOR | Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, Chris Lott |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15364