TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization

Dipkumar Patel · 2026

Read TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization by Dipkumar Patel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27467.txt.

Introduction

Patel shrinks the memory cache that language models keep while writing. His TurboAngle method stores cache entries as angles. A random rotation spreads value pairs near-uniformly on a circle. Each layer can use its own codebook sizes. Key layers get more bits than the rest. Tests span seven models from one to seven billion parameters. Four models show no quality loss at all. Six of seven stay near-lossless. Total cost lands at 6.56 bits per element on Mistral-7B.

Catalog record

CREATORDipkumar Patel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30800
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization Dipkumar Patel Abstract We compress KV cache entries by quantizing angles in the Fast Walsh-Hadamard domain, where a random ± 1 \pm 1 diagonal rotation makes consecutive element pairs uniformly distributed on the unit circle. We extend this angular quantizer with per-layer early-boost : independently configuring K and V codebook sizes at each layer, with higher precision for a model-specific subset of critical layers. Across seven models (1B to 7B parameters), per-layer early-boost achieves lossless compression ( Δ ​ PPL ≤ 0 \Delta\mathrm{PPL}\leq 0 ) on four models and near-lossless quality ( Δ ​ PPL ≤ 0.002 \Delta\mathrm{PPL}\leq 0.002 ) on two more, at 3.28 to 3.67 angle bits per element. Adding norm quantization with asymmetric K/V bit allocation (8-bit linear K norms, 4-bit log-space V norms) yields an end-to-end rate of 6.56 total bits on Mistral-7B with only Δ ​ PPL = + 0.0014 \Delta\mathrm{PPL}={+}0.0014 , requiring zero calibration. A layer-group sensitivity analysis reveals that the critical layers, bottleneck type (K-dominated vs V-dominated), and even the existence of negative-transfer layers where increased precision degrades quality are all model-specific, providing actionable rules for configuring the quantizer on new architectures. 1 Introduction KV cache memory scales as O(LHTd) for a transformer with L layers, H attention heads, head dimension d , and T cached tokens. At long contexts, KV cache dominates model weight storage, making quantization essential for efficient inference. Existing methods [ 10 , 7 , 13 , 5 , 14 ] apply scalar or vector quantization to raw activations, but KV entries exhibit outliers, channel-dependent scales, and non-Gaussian marginals that complicate uniform quantization. These methods compensate with per-channel calibration, asymmetric codebooks, or fine-grained grouping. TurboAngle takes a different approach: transform the activations into a coordinate system where the distribution is provably uniform, then apply the information-theoretically optimal quantizer (uniform bins) with zero calibration. Applying a random \pm 1 diagonal rotation followed by the normalized FWHT produces output pairs whose angles on S^{1} are uniformly distributed in the large- d limit. The simplest possible quantizer is also the optimal one. The uniform-angle approach is effective but treats all layers identically. Transformers do not have uniform layer sensitivity: early layers typically encode broad contextual features that are more sensitive to quantization error, while later layers can tolerate coarser precision. We exploit this by introducing per-layer MixedKV , which assigns independent K-cache and V-cache codebook sizes to each layer. Contributions. • We show that FWHT with random sign rotation produces uniform angles on S^{1} for consecutive element pairs, and build TurboAngle, an angular quantizer that exploits this property at \tfrac{\log_{2}n}{2} bits per element. On Mistral-7B at 3.0 angle bits, TurboAngle achieves 14.8\times lower perplexity degradation than TurboQuant [ 13 ] sym4-g4 at 4.0 bits. • We introduce per-layer MixedKV early-boost: assigning higher angular precision to the first n_{\mathrm{early}} layers (or model-specific critical layer groups) while keeping remaining layers at baseline. This achieves lossless compression ( \Delta\mathrm{PPL}\leq 0 ) on 4 of 7 models and near-lossless ( \Delta\mathrm{PPL}\leq 0.002 ) on 6 of 7, at 3.28–3.67 angle bits. • We characterize per-model sensitivity patterns across seven architectures, discovering K-dominated vs V-dominated bottlenecks, non-monotonic layer-count scaling, and negative-transfer layers where boosting precision actively degrades quality. • We quantize the per-pair norms with asymmetric K/V bit allocation, finding that K norms require 8-bit precision while V norms tolerate 4-bit log-space quantization. The best end-to-end configuration on Mistral-7B ( d=128 ) achieves 6.56 total bits at \Delta\mathrm{PPL}={+}0.0014 with zero calibration. 2 Background Fast Walsh-Hadamard Transform. The normalized Hadamard matrix H\in\{+\tfrac{1}{\sqrt{d}},-\tfrac{1}{\sqrt{d}}\}^{d\times d} defines an orthogonal transform computable in O(d\log d) via a butterfly decomposition. Because H is symmetric and orthonormal, it is self-inverse: H^{-1}=H^{T}=H . The forward and inverse transforms are identical, and the transform preserves norms. Angle uniformity after random rotation. Let D=\mathrm{diag}(s_{1},\ldots,s_{d}) with s_{i}\sim\mathrm{Uniform}(\{+1,-1\}) drawn independently, and define y=HDx for an input x\in\mathbb{R}^{d} . Each output coordinate y_{j}=\tfrac{1}{\sqrt{d}}\sum_{i}s_{i}H_{ji}x_{i} is a weighted sum of d independent sign-randomized terms. As d grows, the Central Limit Theorem drives y_{j} toward a Gaussian. The consecutive pair (y_{2i},y_{2i+1}) approaches a spherically symmetric 2D Gaussian \mathcal{N}(0,\sigma^{2}I_{2}) , because the random diagonal D breaks the inter-coordinate correlations that would otherwise arise from Hadamard structure. For any spherically symmetric 2D distribution, the angle \theta=\mathrm{atan2}(y_{2i+1},y_{2i}) is exactly \mathrm{Uniform}([0,2\pi)) , independent of the radius r=\sqrt{y_{2i}^{2}+y_{2i+1}^{2}} . At d=128 (Mistral-7B’s head dimension), the Gaussian approximation is already tight, and angular uniformity holds empirically to high precision. At d=64 (used by TinyLlama, SmolLM2, OLMo, phi-1.5, and StableLM-2), the approximation remains effective for practical purposes, as confirmed by our experiments. 3 Method 3.1 Angular Quantization TurboAngle encodes each KV cache vector by transforming it into the Hadamard domain with a random sign rotation, decomposing consecutive output pairs into polar coordinates, quantizing the angles uniformly, and storing the norms separately. Algorithm 1 states the compression path. Figure 1 shows the full encode-decode pipeline. Algorithm 1 TurboAngle Encode 0: KV cache tensor x\in\mathbb{R}^{d} , number of angle bins n , rotation matrix D (shared) 1: y\leftarrow H\cdot D\cdot x {normalized FWHT after \pm 1 diagonal rotation} 2: for i=0 to d/2-1 do 3: r_{i}\leftarrow\sqrt{y_{2i}^{2}+y_{2i+1}^{2}} 4: \theta_{i}\leftarrow\mathrm{atan2}(y_{2i+1},\,y_{2i}) 5: k_{i}\leftarrow\lfloor n\cdot\theta_{i}/(2\pi)\rceil\bmod n {uniform angular quantization} 6: end for 7: return \{(r_{i},k_{i})\}_{i=0}^{d/2-1} Figure 1: TurboAngle pipeline. Top: the compression path applies a random diagonal rotation D , the normalized FWHT H , polar decomposition of consecutive pairs, and uniform angle quantization on S^{1} , storing angle indices k_{i} and norms r_{i} . Bottom: reconstruction maps (k_{i},r_{i}) back to Cartesian coordinates via trigonometric lookup, then applies the inverse FWHT to recover the approximate KV vector. Reconstruction maps each stored pair (r_{i},k_{i}) back to Cartesian coordinates: \hat{y}_{2i}=r_{i}\cos(2\pi k_{i}/n) , \hat{y}_{2i+1}=r_{i}\sin(2\pi k_{i}/n) . The original-domain approximation follows from the inverse transform \hat{x}=DH\hat{y} , using the self-inverse property H^{-1}=H and D^{-1}=D . Rate accounting. Each angle index k_{i}\in\{0,\ldots,n{-}1\} requires \log_{2}n bits. With one index per pair of elements, the angular bit rate is \tfrac{\log_{2}n}{2} bits per element. These rates count only angle storage; each pair norm r_{i} is stored in fp32 (equivalently 16 bits per element). Implementation. The diagonal D is sampled once from a seeded PRNG and shared across all layers, heads, and tokens. The FWHT operates head-dimension-wise using in-place butterfly operations in PyTorch, adding negligible latency relative to attention. 3.2 Per-Layer MixedKV Early-Boost Uniform angular quantization applies the same codebook size n to every layer and to both key and value caches. We relax both constraints. Per-layer MixedKV assigns an independent pair (n_{K}^{(\ell)},n_{V}^{(\ell)}) of angle codebook sizes to layer \ell , where n_{K}^{(\ell)} controls key precision and n_{V}^{(\ell)} controls value precision. The average angle bit rate across L layers is: \bar{b}=\frac{1}{L}\sum_{\ell=1}^{L}\frac{\log_{2}n_{K}^{(\ell)}+\log_{2}n_{V}^{(\ell)}}{4} (1) where the factor of 4 accounts for the pair-to-element ratio ( \div 2 ) and the K/V average ( \div 2 ). The simplest and most effective allocation strategy is early-boost : assign higher precision to the first n_{\mathrm{early}} layers while keeping the rest at the uniform baseline ( n_{K}=128,n_{V}=64 , i.e., 3.25 bits). A typical early-boost configuration uses (n_{K}^{(\ell)},n_{V}^{(\ell)})=(256,128) for \ell<n_{\mathrm{early}} , adding approximately 0.5 bits per element to those layers. Not all models respond to simple early-boost. On phi-1.5, we find that a selective configuration is necessary: boosting layers 0–7 and 16–23

Read the complete text