FPTQuant: Function-Preserving Transforms for LLM Quantization

Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel · 2025

Read FPTQuant: Function-Preserving Transforms for LLM Quantization by Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04985.txt.

Introduction

Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, and Markus Nagel present FPTQuant, in a 2025 preprint, a way to shrink large language models to four-bit numbers without breaking them. Quantization saves compute and energy, but outlier values wreck accuracy when applied naively. Their method inserts transforms that leave the model's function unchanged. Those transforms reshape the passing activations into tamer ranges. Three transforms are introduced: two merge into the weights for query, key, and value paths, and one cheaply rescales each token on the fly.

Training tunes the transforms locally and end to end, matching the full-precision model's outputs. No custom kernels are needed, and inference overhead is nearly zero. Static four-bit quantization then runs up to about four times faster than full precision. Accuracy stays on par with or above prior methods, one of which is slower. The work comes from an industry research team.

Catalog record

CREATORBoris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04985

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FPTQuant: Function-Preserving Transforms for LLM Quantization Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel Abstract Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces four novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, (3) a mergeable scaling transform within the MLP block, and (4) a cheap, dynamic scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model’s function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9 × 3.9\times over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off – it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower. 1 Introduction Motivation. Inference on large language models (LLMs) incurs a significant compute toll for every token generated, which ultimately costs money and consumes environmental resources. These costs limit the proliferation of LLM use cases, especially on resource constrained edge devices. They’re also a significant barrier to furthering AI research and democratization. Therefore, improving LLM inference efficiency is a critical goal. Of all the numerous LLM efficiency techniques proposed to date, quantization is by far the most successful; significantly reducing the inference cost by reducing the data bit width across the model. However, compared to previous generations of neural networks, such as CNNs, the transformer architecture used in LLMs is significantly more challenging to quantize. Transforms for aggressive quantization. Outliers in transformer weights, activations, and key-value data are a key limitation on quantization [ 1 , 2 , 3 , 4 , 5 ] . The fundamental issue is that quantizing outliers to a regular grid leads to an unfortunate range-precision trade-off. We can either (1) capture the outliers by increasing the range, but lose valuable precision at the highest distribution density around zero, or (2) retain precision, but have to clip the outliers. Both options unfortunately impact model performance. Prior work has explored operations, such as scalings or rotations, that can be added or applied to pretrained networks to smooth outliers without altering the overall model behaviour in the absence of quantization . For example, Xiao et al. 2024 take a single linear layer, \bm{\mathrm{W}} , with input, \bm{\mathrm{X}} , and apply a per-channel scaling \bm{\mathrm{T}}=\diag(\mathbf{s}) to X before quantizing, to reduce outliers, applying the inverse scales to the linear weights. Without quantizers, (\bm{\mathrm{X}}\bm{\mathrm{T}})(\bm{\mathrm{T}}^{-1}\bm{\mathrm{W}})=\bm{\mathrm{X}}\bm{\mathrm{W}} , but with quantizers Q , Q(\bm{\mathrm{X}}\bm{\mathrm{T}})Q(\bm{\mathrm{T}}^{-1}\bm{\mathrm{W}})\neq Q(\bm{\mathrm{X}})Q(\bm{\mathrm{W}}) . We refer to such operations as function-preserving transforms (FPTs) , for which we desire the following properties: P1 Function-preservation. Without any quantization, inserting transform pairs should not change the output (up to computational errors). In practice, this means each FPT typically has an inverse operation. P2 Expressivity. Transforms with a continuous parametrization and more degrees of freedom are desirable. Continuity means transforms can be optimized directly, e.g. using gradient descent. Extra degrees of freedom offer more flexibility for reducing the quantization error. P3 Compute overhead. Depending on the FPT type and location, it may be possible to merge (or ‘fuse’) it into an existing operation in a pretrained model. Non-mergeable FPTs represent a new op in the computational graph, and incur additional overhead, as well as requiring software and/or hardware support. Contributions. Our contributions are threefold: 1. We introduce FPTQuant: Function-Preserving Transforms for Quantization (Figure 1 ). FPTQuant includes four novel FPTs that are designed to be both expressive and cheap. 2. We show FPTQuant enables static INT4 quantization with minimal overhead. This provides a SOTA speed-up of up to 3.9\times over FP. 3. We show FPTQuant has an excellent accuracy-speed trade-off – it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower. Figure 1 : FPTQuant . FPTQuant consists of 6 transform types. (\bm{\mathrm{T}}_{k},\bar{\bm{\mathrm{T}}}_{k}) is a scale-and-rotate transform merged into the query and key weights; (\bm{\mathrm{T}}_{v},\bar{\bm{\mathrm{T}}}_{v}) consists of invertible matrices per head merged into value and output weights; (\bm{\mathrm{T}}_{u},\bm{\mathrm{T}}^{-1}_{u}) is a per-channel scaler merged into up and down projection weights; transforms \{\bm{\mathrm{S}}_{n}\}_{n=1}^{N} ( N=2\times number of transformer blocks for typical LLMs) are per-token scalers applied to the residual and within the attention and MLP blocks. The scales \bm{\mathrm{S}}_{n} are computed by existing RMSNorms, and in practice means now the RMSNorm is also applied to the residual (versus the original network, see {\color[rgb]{0.7539,0,0}\bm{\times}}^{*} ). We also use partly online Hadamard transform \bm{\mathrm{T}}_{d} [ 7 ] and mergeable rotation matrix \bm{\mathrm{T}}_{r} [ 8 ] . 2 Related work Quantization Neural network quantization has been demonstrated as an effective technique for reducing the model size and improving computational efficiency [ 9 , 10 ] . Quantization methods can generally be categorized into post-training quantization (PTQ) and quantization-aware training (QAT) families. PTQ algorithms take a pretrained high precision network and convert it directly into a fixed-point network without the need for the original training pipeline [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] . These methods are data-free or only require a small calibration dataset, and are generally fast and easy to use. Quantization-aware training (QAT) methods [ 20 , 21 , 22 , 23 , 24 ] simulate quantization during training, allowing the model to find more optimal solutions compared to PTQ. However, they generally require longer training times, increased memory usage, need for labeled data and hyperparameter tuning. LLM quantization The excessive training cost and memory usage of traditional QAT methods make them less suitable for quantizing modern LLMs. A few works focus on developing efficient variants of QAT for LLMs include [ 25 , 26 , 27 , 28 , 29 , 30 ] . Notably, ParetoQ [ 31 ] is the only work we are aware of that scale QAT to billions of tokens. Post-training quantization of LLMs is a challenging task due to presence of strong numerical outliers in weights and activations [ 1 , 2 , 3 , 4 , 5 ] . Various strategies have been explored at tackling these difficulties. These include employing second-order information to mitigate the quantization error [ 32 ] ; emphasizing the importance of so-called “salient” weights that correspond to high-magnitude activations [ 33 , 34 , 35 ] ; separating outliers and use mixed-precision [ 36 , 37 , 38 ] . Some of the other LLM PTQ methods include [ 39 , 40 , 41 , 42 ] . Note that many of these PTQ techniques focus primarily on weight quantization and memory size reduction. Function-preserving

Read the complete text