NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
Donghyun Son, Euntae Choi, Sungjoo Yoo · 2025
Read NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache by Donghyun Son, Euntae Choi, Sungjoo Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18231.txt.
Introduction
Donghyun Son, Euntae Choi, and Sungjoo Yoo shrink the memory cache of large language models without any calibration data, in a paper from 2025. The cache of past keys and values swells with long chats and big batches. Squeezing it into few bits helps, but older squeezers lean on sample data and break when real inputs drift.
Their trick reshapes the numbers before squeezing: normalize each token, shift each channel to center, then normalize again, with a mixing transform on top. The result looks like a standard bell curve, so one shared codebook fits all inputs. Tests at one and two bits beat prior methods and ran up to three times faster than full precision.
Catalog record
| CREATOR | Donghyun Son, Euntae Choi, Sungjoo Yoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18231
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache Donghyun Son, Euntae Choi, Sungjoo Yoo Abstract Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets. To address this limitation, we introduce NSNQuant , a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. By applying a three-step transformation— 1) a token-wise normalization ( N ormalize), 2) a channel-wise centering ( S hift), and 3) a second token-wise normalization ( N ormalize)—with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. This alignment enables robust, calibration-free vector quantization using a single reusable codebook. Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3 × \times throughput gain over full-precision baselines. 1 Introduction Large language models (LLMs) have been widely adopted across various domains due to their strong generalization capabilities ( achiam2023gpt ) . Recently, with the emerging trend of using LLMs to solve complex problems, their use has expanded into long-context scenarios such as long-context reasoning ( wei2022chain ; guo2025deepseek ) and retrieval-augmented generation (RAG) ( lewis2020retrieval ) . However, when processing long sequences, LLM inference requires a significant amount of memory and is memory-bound ( kim2023squeezellm ; hooper2024kvquant ) . One of the major causes is the large size of key-value (KV) cache, which linearly increases with the sequence length. To alleviate this issue, many studies have proposed approaches to compressing KV cache effectively, such as eviction ( zhang2023h2o ; li2024snapkv ; cai2024pyramidkv ; qin2025cake ) and low-rank approximation ( changpalu ; lin2024matryoshkakv ) . Among them, quantization is one of the most widely adopted approaches. Previous studies ( liu2024kivi ; hooper2024kvquant ) analyze how outliers emerge in the KV cache and propose to quantize key and value along the channel dimension and token dimension, respectively. Recently, Coupled Quantization (CQ) ( zhang2024kv ) proposed to quantize multiple channels together, using centroids obtained from the calibration set. The idea of CQ is identical to the concept of vector quantization (VQ), where groups of values are jointly quantized into codebook indices. Utilizing the centroids as codebooks, CQ achieves state-of-the-art performance in diverse tasks, proving the effectiveness of VQ in KV cache quantization. In our study, we observe that CQ fails to generalize in out-of-distribution (OOD) scenarios where the input distribution is different from that of the calibration set (in-distribution) ( yuan2023benchmarking ) . For example, Figure 1(a) shows that while CQ excels in WikiText-2, it performs much worse in C4, since it is calibrated using only a few samples from WikiText-2 dataset. Moreover, Figure 1(b) demonstrates the distribution of key and value can vary greatly by the input distribution. This implies that learning centroids from the small set of data is very risky when applying them to other datasets. To this end, we propose NSNQuant, a calibration-free vector quantization (VQ) method that generalizes well to a wide range of datasets. Our contributions are summarized as follows: • We empirically show that the strong dependence of key-value distributions on the input dataset can lead to severe errors in the existing VQ method. In particular, we observe that the centroids learned by CQ-4c9b fail to accurately quantize important punctuation tokens in LLaMA3-8B and LLaMA3.1-8B on C4, resulting in degraded performance. • To address this limitation, we propose NSNQuant , a calibration-free VQ method for KV cache. NSNQuant effectively matches the key and value distribution with the standard normal distribution through the Normalize-Shift-Normalize (NSN) process and the Hadamard transform, as shown in Figure 3 . Since the approximate distribution of key-value is known prior to the inference, an effective codebook for VQ can be built without any external data. • We conduct comprehensive experiments and analysis to show the effectiveness of NSNQuant across different tasks and models. The results with the LLaMA ( 2024llama3.1 ; touvron2023llama ; grattafiori2024llama ) and Mistral models ( jiang2023mistral7b ) clearly show that NSNQuant outperforms other baselines in 1-bit and 2-bit quantization. Moreover, we implement efficient CUDA kernels for the low-bit computation, improving throughput and reducing memory usage. (a) (b) Figure 1 : (a) PPL evaluation results with LLaMA3.1-8B. Although CQ achieves lower PPL in WikiText-2 (in-distribution), it performs worse on C4 (out-of-distribution). (b) t-SNE visualization of LLaMA3.1-8B key and value. The clustering pattern shows that key and value distributions strongly depend on the input data. More visualization can be found in Figure 11 2 Preliminary LLM inference and KV cache LLM inference has two main stages: prefilling and decoding. In the prefilling stage, all prompt tokens are processed simultaneously by the transformer decoder layers ( vaswani2017attention ) . In the decoding stage, new tokens are generated one by one in an autoregressive manner. In both stages, each token only attends to previous tokens due to the causal nature of masked self-attention. To avoid redundant computation, a KV cache stores key-value pairs from previous tokens. It is initialized during prefilling and updated at each decoding step by appending the latest key-value pair. It becomes a primary bottleneck when processing long sequences, as its size linearly increases with the sequence length. Vector quantization Unlike scalar quantization (SQ) where each scalar value is quantized individually, vector quantization (VQ) compresses a group of values jointly using a codebook. In VQ, a d -dimensional vector is matched to the closest entry in a codebook, and quantized as follows: \operatorname{VQ}(v)=\operatorname*{argmin}_{i}\operatorname{D}(v,\mathbb{C}[i]), (1) where \mathbb{C} denotes the codebook and \operatorname{D}(a,b) is a distance function between vectors a and b . Hadamard transform A Hadamard matrix is an orthogonal matrix whose entries all have the same magnitude. Its size can be doubled recursively through Sylvester’s construction, which builds a larger Hadamard matrix by combining copies of a smaller one in a specific pattern. This recursive definition forms the basis of the Walsh–Hadamard transform, allowing matrix–vector multiplication in O(d\log d) time. QuIP# ( pmlr-v235-tseng24a ) adopts the randomized Hadamard transform (RHT) ( halko2011finding ) , where the sign of each row and column is flipped independently with probability 1/2. We also employ RHT to compute the theoretical bounds presented in Lemma 1 . Figure 2 : Overall structure of attention under NSNQuant. Residuals are omitted for simplicity. We use superscript k and v to mark values associated with key and value, respectively. Since NSN is applied to keys before RoPE, two branches are needed to correctly compute attention scores. See Appendix B for details on attention output computation. Figure 3 : A visual illustration of the effect of NSN on the channel distribution. Our Normalize-Shift-Normalize (NSN) process aligns the channel distribution with the standard normal distribution, when used with the Hadamard transform. 3 Method 3.1 Motivation As shown in Figure 1(a) , we observe several cases where the existing VQ method, coupled quantization (CQ) ( zhang2024kv ) , suffers from severe performance degradation when tested on OOD datasets. We attribute this to the distribution mismatch between datasets, as visualized