KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
Fei Li Song Liu Weiguo Wu Shiqiang Nie Jinyu Wang · 2025
Read KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache by Fei Li Song Liu Weiguo Wu Shiqiang Nie Jinyu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08018.txt.
Introduction
Fei Li, Song Liu, Weiguo Wu, Shiqiang Nie, and Jinyu Wang present a 2025 paper on shrinking the memory that large language models use while generating text. During generation, models store keys and values for past words, and that cache swells with long inputs. Cutting its precision saves memory, but one fixed setting wastes bits on layers that matter little.
Their method scores each layer by how much its matrices move the model's loss, using gradients. Important layers keep more bits; minor ones are squeezed harder. For long texts, recent key words stay at full precision while older ones are compressed more. The balance between accuracy, memory, and speed can be tuned to the hardware at hand.
Catalog record
| CREATOR | Fei Li Song Liu Weiguo Wu Shiqiang Nie Jinyu Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08018
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache Fei Li Song Liu Weiguo Wu Shiqiang Nie Jinyu Wang Abstract The high memory demands of the Key-Value (KV) Cache during the inference of Large Language Models (LLMs) severely restrict their deployment in resource-constrained platforms. Quantization can effectively alleviate the memory pressure caused by KV Cache. However, existing methods either rely on static one-size-fits-all precision allocation or fail to dynamically prioritize critical KV in long-context tasks, forcing memory-accuracy-throughput tradeoffs. In this work, we propose a novel mixed-precision quantization method for KV Cache named KVmix. KVmix leverages gradient-based importance analysis to evaluate how individual Key and Value projection matrices affect the model loss, enabling layer-specific bit-width allocation for mix-precision quantization. It dynamically prioritizes higher precision for important layers while aggressively quantizing less influential ones, achieving a tunable balance between accuracy and efficiency. KVmix also introduces a dynamic long-context optimization strategy that adaptively keeps full-precision KV pairs for recent pivotal tokens and compresses older ones, achieving high-quality sequence generation with low memory usage. Additionally, KVmix provides efficient low-bit quantization and CUDA kernels to optimize computational overhead. On LLMs such as Llama and Mistral, KVmix achieves near-lossless inference performance with extremely low quantization configuration (Key 2.19bit Value 2.38bit), while delivering a remarkable 4.9× memory compression and a 5.3× speedup in inference throughput. 1 Introduction Large Language Models (LLMs) [ 1 ] , such as GPT [ 2 ] , Llama [ 3 ] , and their derivatives, have significantly advanced the field of Natural Language Processing (NLP). These models exhibit outstanding performance [ 4 , 5 ] across a diverse array of tasks, including text generation, question answering, and machine translation. The Key-Value (KV) Cache plays an essential role in the autoregressive decoding process of LLMs. The KV Cache substantially reduces redundant computations in the attention mechanism by storing Key and Value states from preceding time steps for subsequent token generation, thereby significantly accelerating inference efficiency [ 6 ] . Nevertheless, as sequence lengths grow, the memory footprint of the KV Cache increases linearly, presenting a formidable challenge to hardware resources. For example, a model with 70 billion parameters may demand over 50GB of memory to maintain the KV Cache when generating a sequence of 20k tokens—a requirement that far exceeds the memory capacity of typical GPUs. In scenarios involving multiple concurrent requests, the KV Cache for each request cannot be shared due to its dependence on unique preceding prompts. Although model parameters can be reused, memory quickly becomes saturated due to the extensive KV Cache demands. Once memory is depleted, data is offloaded to system memory or even disks, resulting in frequent High Bandwidth Memory (HBM) exchanges with system memory. This process causes latency to surge exponentially, leading to catastrophic performance degradation. The characteristics of the KV Cache outlined above severely restrict the practical deployment and inference efficiency of LLMs in resource-constrained environments, highlighting the pressing need for efficient memory optimization strategies [ 7 , 8 ] . In recent years, research tackling this issue has predominantly focused on reducing the memory overhead of the KV Cache through quantization and sparsification techniques [ 9 , 10 , 11 ] . Quantization methods, in particular, have gained widespread adoption in industry, significantly contributing to the scalability and accessibility of large-scale models [ 12 ] . By quantizing the KV Cache, memory usage can be markedly reduced—for instance, transitioning from FP16 to INT8 halves memory requirements while accelerating memory access and improving throughput. Existing quantization methods have demonstrated impressive model performance even at very low bit-widths. However, these methods either rely on static one-size-fits-all precision allocation schemes [ 13 , 14 ] , lacking flexibility and performance-aware adaptation capabilities, or suffer from the high computational overhead of dynamic quantization while failing to adaptively prioritize critical KV pairs in long-context tasks [ 15 , 16 ] . Therefore, they are forced to make suboptimal trade-offs among memory usage, model accuracy, and computational throughput. To address these problems, this paper proposes KVmix, a novel mixed-precision quantization method for KV Cache. KVmix evaluates KV importance using gradient norms of Key and Value projection weights relative to the model’s loss function, enabling layer-specific mixed-precision quantization with flexible bit-width allocation. It proposes a dynamic long-context optimization strategy, keeping full-precision KV pairs for recent pivotal tokens while compressing older pairs. Additionally, KVmix offers low-bit compression strategies and efficient CUDA implementations. KVmix maximizes KV Cache compression and model throughput while maintaining controlled precision. Our specific contributions are as follows: • We propose a novel layer importance-aware mixed-precision quantization method. This method assesses the importance of KVs at each layer by computing the L2 gradient norms of Key and Value projection weights with respect to the model’s loss function. Based thereon, it independently applies mixed-precision quantization to different layers, allocating higher bit-widths to critical layers and lower to less influential ones. Therefore, it provides the flexibility to balance between accuracy and resource efficiency across diverse inference scenarios. • We propose a dynamic pivotal context selection strategy to optimize long-context tasks. According to the KV importance analysis, it adaptively keeps full-precision KV pairs for recent pivotal tokens while aggressively compressing older pairs. This strategy ensures high-quality sequence generation in long-context inference scenarios while dynamically reducing the number of full-precision KV pairs for better memory optimization. • We design efficient CUDA implementations and a high-compression 3-bit quantization method for KVmix. Extensive experimental results show that KVmix achieves nearly lossless model accuracy across multiple LLMs and datasets, as well as a 4.9 \times reduction in runtime memory usage and a 5.3 \times speedup in inference efficiency, outperforming prior state-of-the-art (SOTA) quantization methods for KV Cache. 2 Related Work and Motivation 2.1 Related Work To mitigate the memory challenges associated with KV Cache, researchers have developed a variety of optimization approaches, primarily centered on compression techniques and dynamic memory management [ 17 , 18 ] . We mainly discuss the KV Cache compression techniques related to this work. Existing compression approaches encompass quantization, sparsification, and KV Cache sharing [ 19 , 20 , 21 ] . Extensive research has focused on reducing the memory overhead of KV Cache through quantization. For example, KIVI [ 13 ] introduced a 2-bit asymmetric quantization technique, employing per-channel quantization for Keys and per-token quantization for Values. This method substantially improves model throughput while preserving accuracy at low bit-widths. KVQuant [ 22 ] proposed a non-uniform quantization strategy, integrating pre-RoPE per-channel Key quantization with per-token Value quantization. It employs offline calibration to manage outliers, achieving robust performance in long-context inference scenarios. QAQ [ 15 ] developed a dynamic mixed-precision quantization method that calculates quantization bits for Keys and Values online and optimizes the trade-off between accuracy and compression ratio by predicting attention scores. Atom [ 23 ] investigated a mixed-precision scheme involving 4-bit and 8-bit activations, dynamically quantizing activations to adapt to input distributions. SKVQ [ 16 ] implemented a sliding window quantization method, maintaining high precision for recent KV pairs while reducing quantization errors through channel reordering. IntactKV [ 14 ] simplified the quantization process by preserving critical tokens (e.g., initial tokens) with full precision, thereby sustaining generation quality. WKVQuant [ 24 ] further compressed the KV Cache by incorporating weight