Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition
Calvin Ang, Sungyoon Kim, Mert Pilanci · 2026
Read Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition by Calvin Ang, Sungyoon Kim, Mert Pilanci (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19559.txt.
Introduction
Ang and co-authors study scalar quantization of two matrices before they are multiplied. Each entry is rounded with its own quantizer, and the product uses the rounded values. They seek the rounding that minimises mean-squared error of the product. In the high-resolution limit, they derive a sharp error expansion and the best centre densities. For correlated Gaussian pairs, the best density has a closed form. A phase transition appears: the density is one-peaked, then two-peaked as correlation grows.
Catalog record
| CREATOR | Calvin Ang, Sungyoon Kim, Mert Pilanci |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59913 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19559
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition Calvin Ang, Sungyoon Kim, Mert Pilanci Abstract We study entrywise scalar quantization of two matrices prior to multiplication. Given A ∈ ℝ m × k A\in\mathbb{R}^{m\times k} and B ∈ ℝ k × n B\in\mathbb{R}^{k\times n} , we quantize entries of A A and B B independently using scalar quantizers with K X K_{X} and K Y K_{Y} levels per entry, and form C ^ = A ^ B ^ \widehat{C}=\widehat{A}\,\widehat{B} . The objective is to minimize the matrix multiplication mean-squared error (MSE) ℰ ≜ 𝔼 [ ‖ A B − A ^ B ^ ‖ F 2 ] \mathcal{E}\triangleq\mathbb{E}[\|{AB-\widehat{A}\widehat{B}}\|_{F}^{2}] under a pair-i.i.d. inner-product model. In the high-resolution regime K X , K Y → ∞ K_{X},K_{Y}\to\infty , we derive a sharp K − 2 K^{-2} asymptotic expansion for ℰ \mathcal{E} , identify the exact optimal leading constants, and characterize asymptotically optimal quantization center densities in terms of conditional second moments. We then specialize to correlated Gaussian multiplicative pairs, obtaining a closed-form optimal point density λ ⋆ ( u ) ∝ exp ( − u 2 6 ) ( ( 1 − ρ 2 ) + ρ 2 u 2 ) 1 / 3 , u = x σ X , \lambda^{\star}(u)\ \propto\ \exp\!\left(-\frac{u^{2}}{6}\right)\bigl((1-\rho^{2})+\rho^{2}u^{2}\bigr)^{1/3},\qquad u=\frac{x}{\sigma_{X}}, with the same form for y / σ Y y/\sigma_{Y} , and prove a correlation-driven phase transition: the density is unimodal at the origin for | ρ | ≤ 1 / 3 \left\lvert\rho\right\rvert\leq 1/\sqrt{3} and becomes bimodal for | ρ | > 1 / 3 \left\lvert\rho\right\rvert>1/\sqrt{3} with peaks at u peak = ± 3 − 1 / ρ 2 u_{\mathrm{peak}}=\pm\sqrt{3-1/\rho^{2}} . We show our method’s applicability in synthetic experiments such as matrix multiplication quantization and least squares optimization, as well as quantization of large language model key and query activations. 1 Introduction Quantized matrix multiplication is central in modern machine learning inference and scientific computing. In many deployments, one does not care about entrywise reconstruction error of operands A and B ; rather, the relevant distortion is the error induced after multiplication 10 . This perspective is closely connected to classical high-rate theory 1 and to the broader quantization literature 4 , which show that optimal quantizer design depends fundamentally on the downstream distortion criterion. At the same time, low-precision quantization has become a practical necessity in modern GPU inference: recent hardware and software stacks increasingly rely on ultra-low-precision formats such as NVFP4 9 , INT8 2 , and FP8 8 to improve throughput, memory efficiency, and energy efficiency for large-scale matrix multiplication workloads arising in large language model inference . These trends make it especially important to understand quantization schemes that are optimal for the product itself, rather than for the separate reconstruction of the input matrices. However, matrix multiplication introduces a distinct bilinear distortion structure: the error contributed by one operand is filtered by the other, and the importance of each entry depends jointly on both factors. Related recent work has studied quantized matrix multiplication in different settings, including nested lattice quantization 10 . In contrast, our focus is on the optimal design of scalar quantizers specifically for matrix multiplication, deriving high-rate laws tailored to product distortion and showing how multiplicative structure and statistical dependence fundamentally change the optimal point density. This paper asks: Which scalar quantizers for entries of A and B minimize the expected Frobenius MSE of the product computed from quantized operands? Contributions. • We derive a high-rate characterization of the optimal achievable matrix multiplication MSE, proving a sharp K^{-2} scaling law with an exact leading constant. • We reduce the matrix objective to two weighted scalar MSE quantization problems driven by conditional second moments, and we identify the optimal companding point densities. • For correlated Gaussian multiplicative pairs, we obtain closed-form optimal densities and prove a correlation-induced unimodal-to-bimodal “phase transition”. • Provide experimental validation of our results on synthetically generated matrices for matrix multiplication and quantized least squares as well as on key and query activations on the GPT-2 and Qwen3 family of models. 2 High-Rate Analysis of Matrix Multiplication MSE This section derives the high-rate reduction from matrix multiplication MSE to two decoupled weighted scalar criteria, and states the resulting sharp K^{-2} constant. All asymptotic steps are justified in Appendix A and Appendix B . 2.1 Problem Formulation Let A\in\mathbb{R}^{m\times k} , B\in\mathbb{R}^{k\times n} , and C=AB . Let Q_{X} and Q_{Y} be scalar quantizers with \left\lvert\mathrm{range}(Q_{X})\right\rvert\leq K_{X} and \left\lvert\mathrm{range}(Q_{Y})\right\rvert\leq K_{Y} . Define \widehat{A}_{i\ell}=Q_{X}(A_{i\ell}),\qquad\widehat{B}_{\ell j}=Q_{Y}(B_{\ell j}),\qquad\widehat{C}=\widehat{A}\,\widehat{B}. We measure performance by the matrix multiplication MSE \mathcal{E}(Q_{X},Q_{Y})\triangleq\mathbb{E}\big[\left\lVert C-\widehat{C}\right\rVert_{\mathrm{F}}^{2}\big]. The quantizer design is governed by the distribution of multiplicative entry-pairs feeding each inner product. We assume each pairs (X,Y) in the matrix multiplication has identical joint distributions. Assumption 1 (Pair-i.i.d. inner products) . For each output entry C_{ij}=\sum_{\ell=1}^{k}A_{i\ell}B_{\ell j} , the pairs \{(A_{i\ell},B_{\ell j})\}_{\ell=1}^{k} are i.i.d. copies of a generic pair (X,Y) with finite moments up to order 4+\epsilon for some \epsilon>0 . Because of this, the entries \{C_{ij}\}_{i\in[m],j\in[n]} are identically distributed (although dependent). Under Assumption 1 , when we denote (X,Y)\sim\mathcal{D} we have \mathcal{E}(Q_{X},Q_{Y})=nm\ \mathbb{E}_{(X_{i},Y_{i})\sim\mathcal{D}}\Big[\big(\sum_{i=1}^{k}X_{i}Y_{i}-\sum_{i=1}^{k}\hat{X}_{i}\hat{Y}_{i}\big)^{2}\Big]. We use the standard fixed-rate identification R_{X}\triangleq\log_{2}K_{X},\qquad R_{Y}\triangleq\log_{2}K_{Y}, and analyze the high-resolution regime K_{X},K_{Y}\to\infty (or equivalently, R_{X},R_{Y}\to\infty ). 2.2 Sharp High-rate Constant and Optimal Point Densities Let D\triangleq XY-\widehat{X}\widehat{Y} for a single pair (X,Y) . Independence across samples (X_{i},Y_{i}) yields \mathbb{E}_{(X_{i},Y_{i})\sim\mathcal{D}}\big[(\sum_{i=1}^{k}X_{i}Y_{i}-\sum_{i=1}^{k}\hat{X}_{i}\hat{Y}_{i})^{2}\big]=k\,\mathbb{E}[D^{2}]+k(k-1)\mathbb{E}[D]^{2}. (1) Appendix A proves that, for the companding quantizers used in the high-rate design, \mathbb{E}[D]=O(K_{X}^{-2}+K_{Y}^{-2}) . Consequently the second term in ( 1 ) is o(K_{X}^{-2}+K_{Y}^{-2}) and does not affect the K^{-2} leading constant. Define quantization errors e_{X}\triangleq X-\widehat{X} and e_{Y}\triangleq Y-\widehat{Y} . Then D=XY-\widehat{X}\widehat{Y}=Ye_{X}+Xe_{Y}-e_{X}e_{Y}. (2) Squaring produces \displaystyle D^{2} \displaystyle=Y^{2}e_{X}^{2}+X^{2}e_{Y}^{2}+e_{X}^{2}e_{Y}^{2}+2XYe_{X}e_{Y}-2Ye_{X}^{2}e_{Y}-2Xe_{X}e_{Y}^{2}. (3) Appendix A shows that the mixed terms on the second line have expectation O(K_{X}^{-2}K_{Y}^{-2}) for the companding quantizers used in the high-rate design. Since K_{X}^{-2}K_{Y}^{-2}=o(K_{X}^{-2}+K_{Y}^{-2}) as K_{X},K_{Y}\to\infty , we obtain the sharp leading reduction \mathbb{E}[D^{2}]=\mathbb{E}[Y^{2}e_{X}^{2}]+\mathbb{E}[X^{2}e_{Y}^{2}]+o(K_{X}^{-2}+K_{Y}^{-2}). (4) Because e_{X} is a deterministic function of X and e_{Y} is a deterministic function of Y , conditioning yields \mathbb{E}[Y^{2}e_{X}^{2}]=\mathbb{E}\!\left[\mathbb{E}[Y^{2}\mid X]\ e_{X}^{2}\right],\qquad\mathbb{E}[X^{2}e_{Y}^{2}]=\mathbb{E}\!\left[\mathbb{E}[X^{2}\mid Y]\ e_{Y}^{2}\right]. Define the conditional second moments as w_{X}(x)\triangleq\mathbb{E}[Y^{2}\mid X=x],\qquad w_{Y}(y)\triangleq\mathbb{E}[X^{2}\mid Y=y]. (5) Then ( 1 ) and ( 4 ), along with the fact that \mathbb{E}[D]=O(K_{X}^{-2}+K_{Y}^{-2}) imply the two-scalar high-rate reduction \displaystyle\mathcal{E}(Q_{X},Q_{Y}) \displaystyle=mnk\Big(\mathbb{E}[w_{X}(X)(X-Q_{X}(X))^{2}]+\mathbb{E}[w_{Y}(Y)(Y-Q_{Y}(Y))^{2}]\Big) \displaystyle\quad+o(K_{X}^{-2}+K_{Y}^{-2}). (6) Now let’s denote f_{X} and f_{Y} the marginal densities of X and Y , and further assume a regularity condition. These conditions are used to justify the asymptotic analysis and to derive weighted optimal rates. Assumption 2 (High-rate regularity) . The pair (X,Y) admits a joint density f_{X,Y} that is twice continuously differentiable. The functions f_{X},f_{Y},w_{X},w_{Y} are continuous and strictly positive on \mathbb{R} , and \int_{\mathbb{R}}(f_{X}(x)w_{X}(x))^{1/3}\,\mathrm{d}x<\infty,\qquad\int_{\mathbb{R}}(f_{Y}(y)w_{Y}(y))^{1/3}\,\mathrm{d}y<\infty. Moreover, the conditional mean functions \mu_{Y|X}(x)\triangleq\mathbb{E}[Y\mid X=x] and \mu_{X|Y}(y)\triangleq\mathbb{E}[X\mid Y=y] are continuously differentiable and the integrability conditions used in Appendix A and Appendix B hold. 1 1 1 These conditions are mild for common smooth models (e.g., jointly Gaussian pairs) and are stated explicitly where they are used. Now denote the integrals in 2 as I_{X}\triangleq\int_{-\infty}^{\infty}(f_{X}(x)w_{X}(x))^{1/3}\,\mathrm{d}x,\qquad I_{Y}\triangleq\int_{-\infty}^{\infty}(f_{Y}(y)w_{Y}(y))^{1/3}\,\mathrm{d}y. (7) An application of Cauchy-Schwartz gives the optimal high-rate quantizer for matrix multiplication. Theorem 1 (High-rate optimal matrix multiplication MSE) . Under Assumptions 1 – 2 , \inf_{Q_{X},Q_{Y}}\ \mathcal{E}(Q_{X},Q_{Y})=mnk\left(\frac{I_{X}^{3}}{12\,K_{X}^{2}}+\frac{I_{Y}^{3}}{12\,K_{Y}^{2}}\right)+o\!\left(\frac{1}{K_{X}^{2}}+\frac{1}{K_{Y}^{2}}\right). (8) Moreover, asymptotically optimal K_{X} - and