Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, Dan Alistarh · 2025
Read Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models by Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, Dan Alistarh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19392.txt.
Introduction
Alina Shutova and co-authors shrink the memory cache that language models keep while writing, in a 2025 paper. For long texts, that cache can fill tens of gigabytes, since it stores a vector for every token and layer. Squeezing the cache usually costs accuracy at high squeeze rates. The team exploits links between cached keys and values across layers to do better.
Their method, AQUA-KV, learns small adapters that predict much of the cache from what remains. The rest is stored in a near-best compact form. On recent Llama models, the cache fell to about two bits per value with almost no loss in quality. Setup takes a few hours on one graphics card, even for the largest models tested. The method is one-shot and simple to apply.
Catalog record
| CREATOR | Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, Dan Alistarh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19392
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, Dan Alistarh Abstract Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning or merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key & Value compression by exploiting two observations: 1) the inherent dependencies between keys and values across different layers, and 2) high-compression mechanisms for internal network states. We propose AQUA-KV, an adaptive quantization for Key-Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to “optimally” compress the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy on state-of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under 1 % 1\% relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: it can be calibrated on a single GPU within 1-6 hours, even for 70B models. † † affiliationnotice: Equal contribution 1 Introduction Figure 1: Comparison of AQUA-KV to alternative Key-Value Cache compression methods for Llama 3.x models in terms of average LongBench score on 14 english tasks (see Section 4 ). Large Language Models (LLMs) are revolutionizing natural language processing, but come with major computational costs, in particular due to the input-quadratic complexity of attention-based Transformer models ( Vaswani 2017 ) . To achieve faster inference and avoid wasteful recomputation of attention scores during autoregressive generation, KV caching is typically employed, where keys and values are saved for later use. Unfortunately, KV-caching comes with its own pitfalls: KV caches are large, especially when handling long sequences ( Bai et al. 2023 ; Xiao et al. 2023 ) . Thus, the memory footprint of a full-length Key-Value cache can reach tens of gigabytes of device memory, sometimes more than the model itself 1 1 1 For the popular Llama 3.2 3B model ( Dubey et al. 2024 ) with a maximum context length of 2^{17} tokens ( {\approx} 131K), the 16-bit cache takes up 15\text{\,}\mathrm{GB} per sequence. For Llama 3.1 70B and Qwen 2.5 72B, it is 42.9\text{\,}\mathrm{GB} per sequence. . This massive memory consumption increases the cost of deployment, and also slows down inference, as the whole process can become memory-bound for large caches ( Hooper et al. 2024 ) . Previous work has proposed methods to compress KV caches using various methods such as quantization and pruning ( Li et al. 2024a ) , that can significantly reduce the memory footprint of KV caches. Yet, as we increase the degree of compression, e.g. 2 bits per value, existing compression techniques begin to lose significant information, resulting in poor accuracy ( Li et al. 2024c ) . In this work, we aim to improve KV cache compression by taking advantage of the inherent structure and dependencies in the cache tensors. Specifically, we analyze the Key-Value cache behavior for state-of-the-art LLMs and find several strong inter-dependencies 1) between cached vectors from adjacent layers, but also 2) between the keys and values within one layer. Starting from these observations, we formulate a practical compression algorithm that explicitly leverages these inter-dependencies, by training compact linear predictors that capture mutual information between cache components. Further, to offset prediction errors, we use data-free vector quantization to achieve a superior compression-accuracy trade-off for the same bit-width. Our method requires only minimal calibration, is compatible with arbitrary quantization schemes and can be further combined with orthogonal compression techniques such as pruning. In summary, our contributions are as follows: 1. We analyze the structure of key-value caches in modern LLMs and highlight several sources of mutual information that can be leveraged for compression. 2. We propose AQUA-KV — a novel compression framework that exploits inter- and intra-layer dependencies to improve quantization accuracy. AQUA-KV works in one shot, based on a lightweight calibration procedure, and shows competitive size-accuracy trade-offs. Additionally, it is compatible with arbitrary quantization techniques, and can be combined with additional compression, such as pruning. 3. We validate the effectiveness of AQUA-KV on modern LLM families in terms of both perplexity and zero-shot accuracy on long-range benchmarks, where AQUA-KV significantly improves accuracy across model types and bitwdiths, particularly for 2-bit compression. 4. We test AQUA-KV compatibility with various quantization and pruning schemes, from simple uniform quantizatin, to modern data-free vector quantizers ( Malinovskii et al. 2024b ) and hybrid quantization & pruning regimes. 5. We develop a reference implementation for AQUA-KV calibration and inference, which is available online 2 2 2 https://github.com/goodevening13/aquakv . 2 Background and Related Work 2.1 KV-Cache Compression So far, the main focus of work on LLM compression has been on the weight quantization , e.g. ( Frantar et al. 2022 ; Lin et al. 2023 ; Tseng et al. 2024a ; Egiazarian et al. 2024 ) . Recently, there has been a growing demand for KV-cache compression, especially in tasks requiring long contexts. The dynamic nature of KV-caching poses unique challenges: while for weights it is acceptable to use “slow but accurate” compression such as codebook-based methods ( Tseng et al. 2024a ; Egiazarian et al. 2024 ) —given that weights are only decoded at inference time—for the KV-cache, both compression and decompression speeds matter , since we are dynamically adding new entries to the cache as well as decoding them at inference time. Another issue is posed by the inherent structure in caches, in particular, the existence of attention sinks ( Xiao et al. 2023 ) and large outlier values ( Liu et al. 2024c ; Hooper et al. 2024 ; Liu et al. 2024a ) , which may not be present in the weights. Next, we detail the main approaches for KV-cache quantization. KV-Cache quantization approaches can be roughly categorized based on quantization granularity and error handling. Quantization Granularity. Several approaches have been developed with varying quantization granularities. For instance, ZipCache ( He et al. 2024 ) and WKVQuant ( Yue et al. 2024 ) implement channel-separable token-wise quantization, while KVQuant ( Hooper et al. 2024 ) and KIVI ( Liu et al. 2024c ) employ a hybrid approach, using per-channel quantization for key tensors while applying per-token quantization for value tensors. QJL ( Zandieh et al. 2024 ) introduces a specialized JL transform for key tensors combined with per-token quantization for value tensors. Methods like MiKV ( Yang et al. 2024b ) , QAQ ( Dong et al. 2024 ) , and SKVQ ( Duanmu et al. 2024 ) employ variable bit widths to balance accuracy and memory reduction. Error Handling. Among the strategies used to address quantization errors, GEAR ( Kang et al. 2024 ) compensates for errors using a low-rank matrix; to handle outliers, which can significantly impact model performance, IntactKV ( Liu et al. 2024b ) maintains full precision for outlier values.