LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
Han Chen & Zining Zhang & Bingsheng He, Zicong Jiang, Pingyi Luo & Mian Lu & Yuqiang Chen · 2025
Read LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation by Han Chen & Zining Zhang & Bingsheng He, Zicong Jiang, Pingyi Luo & Mian Lu & Yuqiang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19950.txt.
Introduction
Han Chen and co-authors shrink the memory that large language models need while generating text, in a 2025 paper. A model's key-value cache stores context for every token, and compressing it to two bits per value would save enormous memory, but earlier methods lost accuracy by guessing which tokens matter. LogQuant instead filters the cache with a logarithmic rule applied across the whole context.
In benchmark tests, the method raised throughput by about twenty-five percent and allowed batch sizes sixty percent larger at the same memory. On hard tasks like math and code completion, its accuracy beat rival techniques by forty to two hundred percent at equal compression. The authors say it drops into popular inference frameworks such as the transformers library with little fuss. The approach targets the memory wall that limits how many users a model can serve at once.
Catalog record
| CREATOR | Han Chen & Zining Zhang & Bingsheng He, Zicong Jiang, Pingyi Luo & Mian Lu & Yuqiang Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19950
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation Han Chen & Zining Zhang & Bingsheng He, Zicong Jiang, Pingyi Luo & Mian Lu & Yuqiang Chen Abstract We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving superior performance. Previous methods either assume that later tokens are more important or attempt to predict important tokens based on earlier attention patterns. Both approaches, however, can result in performance bottlenecks or frequent mispredictions. LogQuant takes a different approach. By applying a log-based filtering mechanism, it selectively compresses the KV Cache across the entire context, achieving better performance with the same or even reduced memory footprint compared to existing methods. In benchmark tests, it enhances throughput by 25% and boosts batch size by 60% without increasing memory consumption. For challenging tasks such as Math and Code Completion, LogQuant improves accuracy by 40% to 200% at the same compression ratio, outperforming comparable techniques. LogQuant integrates effortlessly with popular inference frameworks like Python’s transformers library. Implementation can be available in https://github.com/Concyclics/LogQuantKV . 1 Introduction The rapid evolution of Large Language Models (LLMs) has enabled context window expansion from 4k to 128k tokens ( Meta 2024 ; OpenAI 2024a ) , driving demand for efficient KV cache management in applications like multi-round chatbot conversations ( OpenAI 2024a ; Anthropic 2024 ; DeepSeek 2024 ) and document-based question answering ( Gao et al. 2023 ; Lewis et al. 2020 ) , where comprehensive contextual understanding is required. Moreover, reasoning models such as OpenAI o1 ( OpenAI 2024b ) , increased the demand for even longer reasoning contexts, xacerbated the memory challenges faced in KV cache management. Figure 1: The observed log-distribution pattern is evident not only in the magnitude of attention scores but also in the positions of attention spikes. These spikes become sparser as the model attends to tokens further from the most recent position, indicating that the model not only focuses on nearby tokens. This phenomenon, illustrated here with Llama3-8B-Instruct ( Dubey et al. 2024 ) on the GSM8K dataset ( Cobbe et al. 2021 ) , is consistent across different tasks and models, as further detailed in Section 2 . Recent studies Zhang et al. 2024 ; Li et al. 2024 ; Dong et al. 2024 reveal KV cache’s linear memory growth with context length and even exceeds model weights in long context and batch inference, posing serious deployment challenges. Existing KV Cache compression methods adopt either eviction , (H2O ( Zhang et al. 2024 ) , Keyformer ( Adnan et al. 2024 ) , snapKV ( Li et al. 2024 ) ), aim to reduce memory usage by selectively removing tokens deemed unimportant. or quantization (QAQ ( Dong et al. 2024 ) , KiVi ( Liu et al. 2024c ) ), reduce the precision of less important tokens, retaining more data while minimizing memory costs. Both struggle with importance identification. window-based methods (KiVi, StreamingLLM ( Xiao et al. 2023 ) ) risk missing distant important tokens, while attention-based approaches (H2O, keyformer) suffer prediction errors from historical scores. Our approach addresses these shortcomings by leveraging a key insight: the positions of the attention spikes (i.e. high attention scores) follow a log distribution as shown in Figure 1 , resulting in sparser importance for tokens as they move further from the current position. By utilizing this property, we can outperform existing methods across a wide range of tasks. Additionally, the original absolute positions of KV cache entries can be disregarded without changing the final attention results during the decoding phase, which allows us to enhance the speed of our log-distributed quantization method. The key contributions of this paper are as follows: • Observation of Log-Distributed Attention Spikes : We observe that in various models and downstream tasks, the positions of high attention spikes follow a log distribution, becoming sparser as tokens move further from the current position. This insight underpins our approach to estimate token importance. • Design of LogQuant : Leveraging this log-distribution observation, we introduce LogQuant, a 2-bit quantization technique that significantly improves accuracy. LogQuant outperforms existing methods like KiVi and H2O by better preserving important tokens, achieving a 40% to 200% improvement in accuracy on complex tasks such as Math and Code Completion with the same or higher compression ratio. • Throughput Optimization : By ignoring the absolute positions of KV cache entries, our method further optimizes the speed of quantization/dequantization process without affecting the final attention results, resulting in a 25% increase in throughput and a 60% increase in batch size. The remainder of the paper is organized as follows: Section 2 details the core concepts behind our proposed LogQuant methods, Section 3 present an extensive set of experiments, Section 4 summarizes our findings and discusses potential directions for future work. 2 Methodology In Section 2.1 , we analyze the distribution of attention scores and evaluate the impact of quantization loss, both with and without sink tokens. Section 2.2 explores the distribution of token importance and introduces our log-based selection strategy. In Section 2.3 , we compare the effects of quantization and eviction under this selection scheme, demonstrating the superiority of quantization over eviction. To further enhance efficiency, Section 2.4 prove that attention computation is position-agnostic. Finally, we present the implementation details of our proposed LogQuant method in Section 2.5 . 2.1 Preliminary Study of KV Cache and Attention Scores There are two well-established observations in recent works particularly relevant to KV cache compression. First, many tokens exhibit consistently low attention scores, indicating that their KV cache entries can be safely compressed with minimal impact on performance ( Liu et al. 2024c ) . Second, predicting token importance based on previous decoding steps is unreliable, as attention scores can vary significantly across iterations, making it difficult to accurately identify which tokens should be preserved ( Dong et al. 2024 ; Jiang et al. 2024 ) . This is also demonstrated in Figure 2 . Figure 2: The maximum attention score of each token position across four consecutive decoding steps, marking the high attention positions for illustrating the unpredictable nature of attention scores. This analysis was conducted using Llama3-8B-Instruct ( Dubey et al. 2024 ) on the GSM8K ( Cobbe et al. 2021 ) and OpenBookQA ( Mihaylov et al. 2018 ) datasets. Inspired by the observation of sink tokens ( Xiao et al. 2023 ) , which are the first few tokens that consistently receive high attention scores (Figure 3 ), we included these tokens in the set maintained at original precision to improve accuracy in 2-bit quantization. However, as shown in Table 1 , this adjustment yielded minimal improvement. This suggests that while sink tokens play a role in defining the conversational context, maintaining high precision for only these tokens is insufficient, indicating that tokens beyond the first few are also crucial for preserving model performance. Figure 3: Attention distribution across different token positions, represented as boxplots based on 25% quantiles across all attention heads. The median and overall distribution of attention scores for sink tokens ( Xiao et al. 2023 ) (tokens 0 and 1) are greater than the sum of the most