Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant

Paolo D’Alberto · 2026

Read Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant by Paolo D’Alberto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08114.txt.

Introduction

Paolo D'Alberto studies how to shrink the key-value cache of a language model. The goal is to save memory without wrecking its answers. Three schemes are compared under the same bit budget. One is a plain scalar baseline. One treats keys and values in unlike ways. One applies the full transform pipeline to both.

D'Alberto starts from the geometry of vectors on a sphere. He traces how a projection step on keys inflates inner-product variance. The softmax then amplifies that growth. In tests, the mixed scheme wins at the most common budget on every measure. Treating keys like values is worse at all budgets. A crossover in key error at other budgets adds nuance.

Catalog record

CREATORPaolo D’Alberto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant Paolo D’Alberto Abstract We analyse three KV cache quantization schemes under a fair bit budget: KV (scalar MSE baseline), KQV (WHT + MSE on K K ; WHT + MSE + QJL on V V ), and QKQV (WHT + MSE + QJL on both). Starting from the Beta distribution on the hypersphere, we trace how QJL on K K inflates inner product variance by π / 2 \pi/2 , which softmax amplifies nonlinearly via Jensen’s inequality. Three empirical findings emerge. (1) At n = 4 n=4 (the practically dominant budget), KQV wins on every measure — KL divergence, geometric K K error, and 6D distance — across all distributions and ranks tested. (2) The K–V asymmetry is unconditional: QKQV is consistently worse than KQV in KL divergence at every budget and distribution. (3) A budget-dependent crossover exists: QKQV achieves better geometric K K reconstruction at n ∈ { 2 , 3 , 5 } n\in\{2,3,5\} , KQV at n ∈ { 4 , 6 } n\in\{4,6\} , invariant to rank and tail weight — an open rate-distortion problem. KL ( p ref ∥ p quant ) \mathrm{KL}(p_{\mathrm{ref}}\|p_{\mathrm{quant}}) , K-only by construction, bridges K direction error to routing corruption and output collapse. At n = 4 n=4 , KQV achieves 2.25 × 2.25\times lower KL than QKQV (MW r = − 0.983 r=-0.983 , p < 10 − 23 p<10^{-23} ), explaining the crossover. We introduce a 6D error framework providing distributional discrimination and geometric resolution beyond scalar metrics. 1 The Beta Distribution on the Hypersphere Let \mathbf{x} be uniformly distributed on \mathcal{S}^{d-1} . Each coordinate x_{j} has density f_{X}(t)=\frac{\Gamma(d/2)}{\sqrt{\pi}\,\Gamma\!\left(\tfrac{d-1}{2}\right)}\left(1-t^{2}\right)^{\frac{d-3}{2}},\quad t\in[-1,1], (1) which is Beta \!\left(\tfrac{d-1}{2},\tfrac{d-1}{2}\right) on [-1,1] . As d\to\infty , this converges to \mathcal{N}(0,1/d) . A geometric proof follows. Fix x_{1}=t . The remaining coordinates satisfy x_{2}^{2}+\cdots+x_{d}^{2}=1-t^{2} , forming a (d-2) -sphere of radius \sqrt{1-t^{2}} in \mathbb{R}^{d-1} . The surface area element at value t is \begin{split}|dS\big|_{t}&\equiv|\mathcal{S}^{d-2}|\cdot(1-t^{2})^{\frac{d-2}{2}}\cdot\frac{dt}{\sqrt{1-t^{2}}}\\ &=|\mathcal{S}^{d-2}|\cdot(1-t^{2})^{\frac{d-3}{2}}\,dt,\end{split} (2) where the factor 1/\sqrt{1-t^{2}} is the arc-length correction (the sphere curves away from the vertical axis). Normalising by |\mathcal{S}^{d-1}| and substituting |\mathcal{S}^{k}|=2\pi^{(k+1)/2}/\Gamma\!\left(\tfrac{k+1}{2}\right) yield Equation ( 1 ). Both surface areas are expressed via Gamma functions: |\mathcal{S}^{d-1}|=2\pi^{d/2}/\Gamma(d/2) and |\mathcal{S}^{d-2}|=2\pi^{(d-1)/2}/\Gamma\!\left(\tfrac{d-1}{2}\right) . Their ratio is the normalisation constant in Equation ( 1 ). The Gamma function arises naturally from the Gaussian radial integral \int_{0}^{\infty}r^{d-1}e^{-r^{2}}dr=\Gamma(d/2)/2 , which is the universal size unit for d -dimensional sphere geometry. Each time a coordinate is fixed and the sphere projected, the dimension drops by one and the Gamma argument shifts by 1/2 : from d/2 to (d-1)/2 . The density ( 1 ) is precisely that dimensional accounting. For large d and small t , (1-t^{2})^{(d-3)/2}\approx e^{-dt^{2}/2} , a Gaussian with variance 1/d . This convergence is empirically visible: the coordinate histogram is U-shaped (arcsine) at d=2 , a mild arch at d=4 , a bell at d=8 , and indistinguishable from Gaussian at d=1024 . Figures 1 , 2 , 4 , and 7 show this progression across four representative dimensions. Notice that the Beta distribution is defined in the interval [-1,1] and the Gaussian for any finite d is not and it can only be approximated to. Figure 1 : d=2 , Beta (0.5,0.5) — arcsine (U-shaped). Top : rotated coordinate histograms (blue) vs. Beta density (red) for gaussian , heavy_tail , low_rank . Bottom : KL divergence boxplots, MSE 4-bit vs. MSE 3-bit + QJL 1-bit (equal budget). MSE wins across all distributions. Circles denote outliers beyond 1.5\times\mathrm{IQR} ( {\approx}2.7\sigma for a Gaussian). 2 The TurboQuant Pipeline In transformer attention, the key and value tensors computed at each decoding step are stored and reused for all future steps; this memory store is the KV cache . Its size grows linearly with context length, making it the dominant memory bottleneck at inference time and the primary target for compression. TurboQuant [ 1 ] compresses KV cache vectors through a two-stage pipeline. The first stage applies a structured random rotation followed by a Beta-optimal scalar quantiser; the second stage corrects the inner product bias in the residual via a 1-bit QJL sketch. Together the two stages target orthogonal problems — variance and bias — as we describe in turn. 2.1 Stage 1: Rotation and Beta-Optimal Quantisation A random Hadamard–Rademacher transform \mathbf{x}_{\mathrm{rot}}=\tfrac{1}{\sqrt{d}}\,H\,\mathrm{diag}(\mathbf{s})\,\hat{\mathbf{v}} (where \hat{\mathbf{v}}=\mathbf{v}/\|\mathbf{v}\| and \mathbf{s}\in\{-1,+1\}^{d} is a random sign vector) is applied before quantisation. As shown in Section 1 , each coordinate of \mathbf{x}_{\mathrm{rot}} follows the Beta distribution, enabling a scalar Lloyd-Max quantiser [ 3 , 4 ] to be designed once for this known density and applied coordinate-wise. The original norm \|\mathbf{v}\| is stored separately. This minimises the mean squared error (MSE) of the quantised representation. Limitation. The random rotation is a single fixed mapping, not a per-vector fresh rotation. It moves data off the coordinate axes and provides expected uniformity over the randomness of \mathbf{s} , but it cannot make a non-isotropic distribution isotropic. In other words, the Hadamard transform with proper normalisation is an orthonormal transformation: applied to unit vectors on the sphere, it performs a rotation. The randomisation, if independent of the process generating the vectors, is introduced to disrupt pathological axis-aligned patterns (e.g., the vector (1,0,\dots,0) ). For a Gaussian input the Beta guarantee holds; for low-rank or subspace-concentrated inputs the marginal coordinates may appear Beta while the joint structure remains degenerate. We reproduce such pathological but realistic embedding distributions and show where Equation ( 1 ) fails in practice. 2.2 Stage 2: QJL Residual Correction Let \mathbf{r}=\mathbf{x}_{\mathrm{rot}}-\hat{\mathbf{x}}_{\beta} be the Stage 1 residual. The 1-bit QJL sketch [ 2 ] \mathbf{q}=\mathrm{sign}\!\left(H(\mathbf{s}^{\prime}\odot\mathbf{r})\right) with a fresh sign vector \mathbf{s}^{\prime} yields the unbiased estimator \hat{\mathbf{r}}=\frac{\sqrt{\pi/2}\,\|\mathbf{r}\|}{d}\,\mathbf{s}^{\prime}\odot H\mathbf{q},\qquad\mathbb{E}[\langle\mathbf{q},\hat{\mathbf{r}}\rangle]=\langle\mathbf{q},\mathbf{r}\rangle\quad\forall\,\mathbf{q}. (3) The constant \sqrt{\pi/2} corrects for the sign-quantisation bias: for z\sim\mathcal{N}(0,\sigma^{2}) , \mathbb{E}[|z|]=\sigma\sqrt{2/\pi} , so \mathrm{sign}(z) underestimates magnitude by \sqrt{2/\pi} . The unbiasedness of \hat{\mathbf{r}} is unconditional : it holds for any residual vector \mathbf{r} , regardless of whether Stage 1 is optimal. What Stage 1 optimality affects is \|\mathbf{r}\| , and hence the variance of the QJL estimate, which is proportional to \|\mathbf{r}\|^{2}/d . Figure 2 : d=8 , Beta (3.5,3.5) — mild bell. Top : coordinate histograms begin to concentrate near zero; low_rank (rank = 1) shows a nearly flat histogram, deviating from Beta — the first sign of joint-structure failure. Bottom : MSE wins across all distributions under equal budget. Circles denote outliers beyond 1.5\times\mathrm{IQR} ( {\approx}2.7\sigma for a Gaussian). Note that Stage 2 applies the same Hadamard structure as Stage 1, but with a fresh randomisation \mathbf{s}^{\prime} drawn independently of \mathbf{s} , ensuring the sketch is uncorrelated with the quantisation error. 2.3 Three Schemes Under Study All experiments compare three schemes at equal effective bit budget n per cache vector. We adopt the following shorthand throughout: Table 1: The three quantisation schemes compared throughout this paper. Name K cache V cache WHT KV MSE, n bits MSE, n bits No KQV MSE, n bits MSE (n{-}1) bits + 1-bit QJL Yes QKQV MSE (n{-}1) bits + 1-bit QJL MSE (n{-}1) bits + 1-bit QJL Yes KV is the scalar baseline: no rotation, no sketch. KQV applies WHT to both caches, then QJL as a residual corrector on V only — this

Read the complete text