ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
Guangda Liu, Chengwei Li, Jieru Zhao * , Chenqi Zhang, Minyi Guo · 2024
Read ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression by Guangda Liu, Chengwei Li, Jieru Zhao * , Chenqi Zhang, Minyi Guo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03213.txt.
Description
Guangda Liu and four co-authors compress the memory cache of language models without throwing context away. Their paper appeared in two thousand twenty-four. Long documents swell the key-value cache and slow replies. Older compression drops tokens for good, losing facts. Page-based recall keeps tokens but groups them by position alone. Meaning-based grouping is the missing piece.
Their ClusterKV groups tokens into semantic clusters. Recall then fetches whole clusters when later steps need them. Algorithms and systems work make the clustering fast enough to use. Accuracy holds up better than with eviction or page schemes. Long-document question answering is a prime use case. Complex reasoning over long context also benefits. The cache shrinks while the past stays reachable.
Catalog record
| CREATOR | Guangda Liu, Chengwei Li, Jieru Zhao * , Chenqi Zhang, Minyi Guo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03213