Optimal Formats for Weight Quantisation
Douglas Orr, Luka Ribar, Carlo Luschi · 2025
Read Optimal Formats for Weight Quantisation by Douglas Orr, Luka Ribar, Carlo Luschi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12988.txt.
Introduction
Orr, Ribar, and Luschi build a principled way to design number formats for shrinking neural net weights in twenty twenty-five. Quantized nets store weights in few bits to run cheap. Formats are usually picked by trial and error. Their frame ties format choice to classic quantization theory.
Top formats win by using variable length codes, they show. The design goal becomes matching output error under a size cap. Curves for block scaled data beat fixed length forms across data families. A Fisher based rule then deals bits across layers unevenly. It saved up to a quarter bit per weight on large language models.
Catalog record
| CREATOR | Douglas Orr, Luka Ribar, Carlo Luschi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Formats for Weight Quantisation Douglas Orr, Luka Ribar, Carlo Luschi Abstract Weight quantisation is an essential technique for enabling efficient training and deployment of modern deep learning models. However, the recipe book of quantisation formats is large and formats are often chosen empirically. In this paper, we propose a framework for systematic design and analysis of quantisation formats. By connecting the question of format design with the classical quantisation theory, we show that the strong practical performance of popular formats comes from their ability to represent values using variable-length codes . We frame the problem as minimising the KL divergence between original and quantised model outputs under a model size constraint, which can be approximated by minimising the squared quantisation error, a well-studied problem where entropy-constrained quantisers with variable-length codes are optimal. We develop non-linear quantisation curves for block-scaled data across multiple distribution families and observe that these formats, along with sparse outlier formats, consistently outperform fixed-length formats, indicating that they also exploit variable-length encoding. Finally, by using the relationship between the Fisher information and KL divergence, we derive the optimal allocation of bit-widths to individual parameter tensors across the model’s layers, saving up to 0.25 0.25 bits per parameter when applied to large language models. \institution Graphcore Research \gcabstract 1 Introduction Weight quantisation enables large deep learning models to run on low-resource hardware and edge devices by saving space and memory bandwidth usage. It can be seen as an optimisation problem, where the goal is to retain the behaviour of the high-precision reference model while reducing the total number of bits needed to store its parameters. This naturally splits into two sub-problems of format design and quantisation procedure, both of which are highly active areas of research. We focus on the format design question, i.e., how to choose a representation space for model parameters. This is somewhat independent from the quantisation procedure, which aims to find an optimal point in that space. The space of possible formats for a model is rich, where element formats including integer, floating-point and non-uniform quantisers can be combined with tensor, channel or block scaling and augmented using sparse outlier storage or rotations [ 25 , 32 , 31 , 41 , 34 , 47 ] . Since this combinatorial space is too large to be explored directly, empirical studies typically narrow the search space. This work addresses the problem of optimal format design, minimising the KL divergence against a reference model under a memory constraint. Following [ 44 ] , who observe that Fisher information can be used to predict the degradation based on the squared error of quantisation, we employ classical quantisation theory [ 2 , 7 ] to derive optimal elementwise quantisers that minimise squared error. Given this scheme for element formats, we empirically evaluate scaled formats based on the absmax or RMS of the tensor, channel or sub-tensor block, for direct-cast quantisation and quantisation-aware training. Our main takeaway is that the most effective quantisation formats all employ some form of variable-length encoding (see fig. 1 ). Block formats outperform optimal elementwise formats by effectively allocating their scale bits to represent the block maximum. Tensor-scaled formats can be effective if sparse outliers are stored separately, again implying variable bits-per-element. Finally, quantisation followed by lossless compression employs variable length explicitly and doesn’t benefit from block scaling or sparse outlier removal. Contributions Our work explains the performance of various quantisation schemes through the lens of optimising KL divergence and weight statistics. Towards this, we propose \sqrt[3]{\mathrm{p}} block-scaled Normal, Laplace and Student-t non-uniform quantisers. We also propose the signmax scaling scheme. These are, to the best of our knowledge, novel contributions. Outline Section 2 presents our framework: the optimisation problem, assumptions and solutions. Experiments on synthetic data in section 3 and LLM quantisation in section 4 demonstrate that block and sparse outlier formats outperform fixed-length codes. Sections 5 , 6 and 7 present related work, limitations and conclusions. Table 3 provides a reference for our notation. Figure 1 : The trade-off between average bits per parameter and top- k KL divergence for Llama 3.1 8B. To approach optimal performance, some form of variable-length encoding is needed: lossless compression, block (or channel) absmax scaling or 0.1\% sparse outlier removal. The shaded line width is \pm 2 standard error over evaluation data. See fig. 8 for other models. 2 Optimal Quantisation Formats In this section, we present our framework for format design. We begin by defining the overall optimisation problem at the model level, showing how this objective can be reduced to minimising the squared error of individual tensors through appropriate approximations. Next, we present solutions for RMS and absmax scaling schemes, as well as a lossless compression scheme. Finally, we revisit model-level optimisation by proposing a scheme for optimal bit allocation across tensors. Consider a probabilistic model with output y conditioned on input drawn from a dataset x\sim\mathcal{X} , where the pdf \mathrm{p}_{\theta}(y\mid x) is represented by a neural network, parametrised by \theta\in\mathbb{R}^{|\theta|} . We wish to find a quantised parameter vector \tilde{\theta}^{\star} which is in \tilde{\Theta}\subset\mathbb{R}^{|\theta|} , minimising the expected KL divergence of \mathrm{p}_{\tilde{\theta}}(y\mid x) against the reference model, \displaystyle\tilde{\theta}^{\star} \displaystyle\coloneqq\argmin_{\tilde{\theta}\in\tilde{\Theta}}\mathrm{D_{KL}}\left(\mathrm{p}_{\theta}\|\mathrm{p}_{\tilde{\theta}}\right), \displaystyle\mathrm{D_{KL}}\left(\mathrm{p}_{\theta}\|\mathrm{p}_{\tilde{\theta}}\right) \displaystyle\coloneqq\mathop{{}\mathbb{E}}_{x\sim\mathcal{X}}\left[\mathop{{}\mathbb{E}}_{\mathrm{p}_{\theta}(y\mid x)}\left[\log\frac{\mathrm{p}_{\theta}(y\mid x)}{\mathrm{p}_{\tilde{\theta}}(y\mid x)}\right]\right], (1) \displaystyle\text{with }|\tilde{\Theta}| \displaystyle\leq 2^{|\theta|\cdot b}. The final line induces a compression constraint on \tilde{\Theta} , specifying an average of b bits per parameter. Considering \tilde{\theta} close to \theta , we can approximate the objective as \mathrm{D_{KL}}\left(\mathrm{p}_{\theta}\|\mathrm{p}_{\tilde{\theta}}\right)\approx\frac{1}{2}(\tilde{\theta}-\theta)^{\top}F\,(\tilde{\theta}-\theta), (2) where F\in\mathbb{R}^{|\theta|\times|\theta|} is the Fisher information matrix of the reference model (see appendix A for a derivation). As a further simplification, we assume the cross terms are small, in which case the approximation reduces to the squared quantisation error, weighted by the diagonal of the Fisher information matrix. Deep learning model parameters can be partitioned into tensors , often the maximal sets of parameters that can be applied in parallel to the intermediate activations — in other words, where the input to the forward pass operation of any scalar parameter does not depend on any other parameter from the same tensor. It is convenient to encode each tensor using a single format, although different tensors may use different formats, so we expand \tilde{\Theta}=\prod_{t}\tilde{\Theta}_{t} , and \tilde{\theta}_{T_{t}}\in\tilde{\Theta}_{t} where T_{t}\in\mathbb{N}^{|T_{t}|} is a vector of parameter indices belonging to tensor t . For much of our analysis, we further approximate the diagonal of the Fisher matrix as constant ( =\bar{f}_{t} ) within each parameter tensor. With this approximation, the increase in KL divergence due to quantisation is a weighted sum across parameter tensors of the unweighted squared error, \mathrm{D_{KL}}\left(\mathrm{p}_{\theta}\|\mathrm{p}_{\tilde{\theta}}\right)\approx\frac{1}{2}\sum_{t}\bar{f}_{t}\cdot\sum_{i\in T_{t}}(\tilde{\theta_{i}}-\theta_{i})^{2}. (3) This form is convenient, as the squared error is easy to compute and the diagonal of F can be estimated with computational and memory costs comparable to a few steps of SGD. We test this equation for predicting the KL divergence after perturbing with iid normal noise in figs. 11 and 13 . 2.1 Optimal tensor formats for known distributions We now turn to the problem of designing a format to represent a single parameter tensor. The formats we consider all operate on blocks of data — all or part of the tensor. The block size B is fixed within a tensor, but may vary across tensors. For the i th