QwT-v2: Practical, Effective and Efficient Post-Training Quantization

Ningyuan Tang, Minghao Fu, Hao Yu, Jianxin Wu · 2025

Read QwT-v2: Practical, Effective and Efficient Post-Training Quantization by Ningyuan Tang, Minghao Fu, Hao Yu, Jianxin Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20932.txt.

Introduction

Shrinking the numbers of a neural network to fewer bits saves memory and power, but clumsy shrinking costs accuracy. Ningyuan Tang and co-authors present QwT-v2 in a 2025 paper, a post-training method that adds a tiny compensation step. Its parent method worked well yet added bulk and clashed with much hardware.

The new compensation is a light per-channel adjustment with very few extra values to store. It folds into standard inference engines with little effort, so the extra cost nearly vanishes. Accuracy matched or beat the parent method across tests. The authors pitch it as a practical route to small models that run on the chips people already own.

Catalog record

CREATORNingyuan Tang, Minghao Fu, Hao Yu, Jianxin Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20932

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QwT-v2: Practical, Effective and Efficient Post-Training Quantization Ningyuan Tang, Minghao Fu, Hao Yu, Jianxin Wu Abstract Network quantization is arguably one of the most practical network compression approaches for reducing the enormous resource consumption of modern deep neural networks. They usually require diverse and subtle design choices for specific architecture and tasks. Instead, the QwT method is a simple and general approach which introduces lightweight additional structures to improve quantization. But QwT incurs extra parameters and latency. More importantly, QwT is not compatible with many hardware platforms. In this paper, we propose QwT-v2, which not only enjoys all advantages of but also resolves major defects of QwT. By adopting a very lightweight channel-wise affine compensation (CWAC) module, QwT-v2 introduces significantly less extra parameters and computations compared to QwT, and at the same time matches or even outperforms QwT in accuracy. The compensation module of QwT-v2 can be integrated into quantization inference engines with little effort, which not only effectively removes the extra costs but also makes it compatible with most existing hardware platforms. 1 Introduction The rapid scaling of deep neural networks guarantees their strong performance in various fields, including language ( Devlin et al. (2019) ; Brown et al. (2020) ), vision ( Dosovitskiy et al. (2021) ; He et al. (2017) ), speech ( Zhang et al. (2023) ), and multimodal ( Radford et al. (2021) ; Liu et al. (2023) ), but also incurs heavy burden on hardware resources such as compute, storage and energy. Network quantization ( Liang et al. (2021) ) stands out as a highly practical approach to reduce these resource consumptions, highlighting large compression ratio, low accuracy loss, and high hardware friendliness. Existing quantization methods are often categorized into Quantization-Aware Training (QAT) which requires training, and Post-Training Quantization (PTQ) which is training free. Neither PTQ nor QAT is perfect. PTQ ( Li et al. (2021) ; Xiao et al. (2023) ) methods are fast but may lead to considerable accuracy degradation when the bit-width is low, while QAT ( Esser et al. (2020) ; Zhu et al. (2023) ; Li et al. (2024) ) methods are thousands of times slower than PTQ methods. Moreover, existing quantization methods require meticulous design for different models and tasks. Changes in model architecture or task requirements can potentially render these quantization methods ineffective. For example, the issue of handling outliers is not important in quantizing CNNs, but is critical for quantizing Transformers. Quantization without Tears (QwT) by Fu et al. (2025) is a simple and general quantization method which suits various network architectures and applications. By adding external compensation matrices to a quantized network S^{\mathbb{Z}} (i.e., the output of a PTQ method), QwT can effectively improve network accuracy without severely affecting latency or model size. In addition, parameters of compensation modules in QwT can be set in closed-form with a small calibration set. This fact enables QwT to train very quickly without any hyperparameter tuning. But, despite being simple, effective and widely applicable, a few drawbacks in QwT remain, which critically limit its deployment in many other real-world scenarios. 1. Hardware compatibility. QwT requires the inference engine to compute mainly in quantized fixed-point format (e.g., INT8), but there are also floating-point computations (e.g., FP16 or BF16) for the compensation modules. Although this capability is available in many GPUs, it is not yet supported well in many other hardware platforms (e.g., simple AI accelerators in a cellphone or an embedded hardware) or software inference engines like vLLM ( Kwon et al. (2023) ). This incompatibility severely restricts QwT’s usability. 2. Extra parameters. QwT compensates a PTQ model with linear projections. In high-bit quantization, compensation matrices only account for a small percentage of the network size. But in a low-bit quantized network, the floating-point compensation matrices become non-negligible and increase model size by around 30%, as shown by Fu et al. (2025) . 3. Lower efficiency. For each compensated block, QwT receives the block input, calculates floating-point compensation, and adds it to the block output. During this process, the inference engine needs to store all these tensors, which increases both memory footage and latency (due to both more memory accesses and more computations). In this paper, we show that all these issues can be resolved simultaneously. We propose QwT-v2, which inherits the advantages of QwT and also showcases the following improvements over it: • Minimal extra parameter & computation. We introduce a new form of compensation module, named channel-wise affine compensation (CWAC), which requires significantly less parameters and computations than those in QwT. • Easy integration & no floating-point computation. Furthermore, the CWAC module can be integrated into inference engines and then it incurs zero overhead during inference. The integration requires only few efforts—merely changing quantization parameters without changing the computation graph. After this integration, floating-point computations are not required, hence QwT-v2 suits most hardware platforms . • Fine-grained compensation. Thanks to the parameter and computation efficiency of CWAC, we can compensate each quantized module individually, rather than compensating for an entire network block. This guarantees better result of QwT-v2 in many settings. 2 Related Works As aforementioned, network quantization methods can be categorized into two lines: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). QAT methods ( Liu et al. (2020) ; Lee et al. (2021) ; Esser et al. (2020) ; Bhalgat et al. (2020) ; Li et al. (2024) ) simulate low-bit computation during training, and minimize accuracy degradation caused by quantization. LSQ ( Esser et al. (2020) ) updates not only parameters but also scale factors via gradient descent during training. LSQ+ ( Bhalgat et al. (2020) ) extends LSQ to non-uniform quantization. Q-ViT ( Li et al. (2024) ) improves quantization performance on Transformers by rectifying information distortion of quantized self-attention maps. PTQ methods ( Li et al. (2021) ; Yuan et al. (2022) ; Frantar et al. (2023) ) convert a well-trained floating-point model to low-bit fixed-point one with a small calibration set. Several methods ( Nagel et al. (2020) ; Li et al. (2021) ; Wei et al. (2022) ) work well on CNNs but fail to be effective on Transformers, due to different network structure and weight/activation numerical distributions. Methods have also been proposed to quantize Transformers ( Yuan et al. (2022) ; Frantar et al. (2023) ; Li et al. (2023) ; Lin et al. (2022) ; Xiao et al. (2023) ). FQ-ViT ( Lin et al. (2022) ) uses power-of-2 factor and log-int-softmax quantization to tackle the difficulty of quantizing layer norm and softmax activations. RepQ-ViT ( Li et al. (2023) ) transforms complex quantizers to simple quantizers via scale reparameterization for efficient inference. GPTQ ( Frantar et al. (2023) ) exploits a single-shot weight quantization technique which utilizes approximate second-order information. 3 The QwT-v2 Method QwT by Fu et al. (2025) proposes a simple and general quantization method to improve PTQ methods, and it is effective on both CNNs and Transformers. For example, given a Transformer block (i.e., MHSA+FFN) whose output in the floating-point and quantized networks are Y^{\mathrm{full}} and Y^{\mathrm{quant}} , respectively; while its input in the quantized version is X^{\mathrm{quant}} . QwT noticed that quantization leads to the difference between Y^{\mathrm{full}} and Y^{\mathrm{quant}} , and reducing this

Read the complete text