MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization

Le Su, Xing Luo, Zhi Jin · 2026

Read MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization by Le Su, Xing Luo, Zhi Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17997.txt.

Introduction

Le Su and two colleagues propose Module-Adaptive Residual Reconstruction for low-bit quantization. Residual methods add residuals to cut error from prior layers, but these can also add Hessian-approximation bias. The team shows a scaling factor on the residual term cuts this bias while keeping error correction, with module-dependent trade-offs. MARR gives each module a scaling factor to balance correction and bias, using a PID update rule with reconstruction error as feedback. Tests on language models and vision transformers show gains up to 20.2% and 4.6% respectively.

Catalog record

CREATORLe Su, Xing Luo, Zhi Jin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68563
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization Le Su, Xing Luo, Zhi Jin Abstract Recently, residual reconstruction-based model quantization methods have achieved promising performance in low-bit post-training quantization (PTQ) by introducing cross-layer residuals to reduce error accumulated from previous layers. However, these residuals may also introduce additional bias arising from the Hessian-approximation (HA) assumption underlying reconstruction-based PTQ, leading to suboptimal quantization performance. In this work, we analyze that multiplying the residual term by a scaling coefficient provides a direct way to mitigate the HA bias associated with residual strength, while preserving accumulated-error correction. More importantly, we observe that this trade-off is module-dependent, making a single global residual strength insufficient to balance effective correction and residual-related bias across modules. Based on these observations, we propose Module-Adaptive Residual Reconstruction (MARR), which assigns a module-specific scaling coefficient to adaptively balance accumulated-error correction and residual-related HA bias for each module. To avoid expensive per-module coefficient search and obtain a stable coefficient estimate, we design a Proportional-Integral-Derivative (PID)-based adaptive update strategy that uses reconstruction error as feedback to progressively refine this coefficient. Experiments on several typical large language models (LLMs) and vision transformers (ViTs) demonstrate the effectiveness of MARR under low-bit quantization ( ≤ \leq 4-bit), achieving up to 20.2% performance gains on LLMs and up to 4.6% relative gains on ViTs over the residual reconstruction state-of-the-art methods. Code will be made publicly available upon acceptance. 1 Introduction Large language models (LLMs) [ 36 , 41 ] and vision transformers (ViTs) [ 23 , 9 ] have achieved remarkable progress across diverse tasks. However, their large model sizes and dense operations incur substantial memory, bandwidth, and inference overhead. To improve deployment efficiency, existing studies either compress pre-trained models through pruning [ 34 , 6 ] , knowledge distillation [ 45 , 13 ] , low-rank decomposition [ 42 ] , and quantization [ 12 , 46 ] , or directly design lightweight architectures [ 21 ] and efficient operators [ 29 , 15 ] . Among them, model quantization is deployment-friendly since it reduces the numerical precision of model while largely preserving the original architecture. Therefore, it is a plug-and-play compression technique compatible with existing inference pipelines and hardware kernels [ 11 ] . Moreover, pushing quantization below conventional 8-bit settings is increasingly important, as low-bit quantization can further reduce memory and bandwidth costs, allowing for larger models, longer sequences, or higher throughput under the same hardware budget [ 12 ] . Therefore, how to push models to lower bit-widths while preserving their performance remains a valuable and challenging problem. Existing quantization methods are broadly divided into quantization-aware training (QAT) [ 24 ] and post-training quantization (PTQ) [ 18 , 37 ] . Compared with QAT, PTQ avoids additional training or finetuning, making it more practical for extremely large pre-trained models. Current PTQ methods can be roughly grouped into quantization parameter-based methods [ 47 ] , redistribution-based methods [ 40 , 3 ] , and reconstruction-based methods [ 11 , 19 ] . Among them, reconstruction-based methods have attracted increasing attention for large models, since they efficiently reduce quantization error by adjusting weights while maintaining low quantization overhead through closed-form updates. A representative basic reconstruction method is GPTQ [ 11 ] , which extends Optimal Brain Surgeon (OBS) [ 14 ] and Optimal Brain Quantization (OBQ) [ 10 ] by applying Hessian-based column-wise updates for each layer. However, under low-bit quantization, quantization errors from preceding layers accumulate across the network, making local-only basic reconstruction insufficient. To address this issue, recent residual reconstruction-based methods (GPTAQ [ 19 ] and ResComp [ 17 ] ) account for inter-layer activation dependencies by introducing a residual between full-precision (FP) and quantized (Quant) activation, which improves low-bit quantization performance. This cross-layer residual is beneficial for reducing accumulated quantization errors; however, it may also introduce additional Hessian-approximation (HA) bias. Such a two-sided effect makes the overall reconstruction suboptimal when overly strong residual correction is applied. In this work, to address this issue, we introduce a scaling coefficient to modulate the residual term, as the HA bias is directly affected by the strength of the residual correction. By doing so, residual reconstruction can preserve effective cross-layer error compensation while reducing the adverse effect of excessive residual strength. Moreover, the desired modulation is inherently module-dependent: the same residual strength can lead to different reconstruction behaviors across modules. This module-wise heterogeneity makes a single global coefficient insufficient. Accordingly, we propose Module-Adaptive Residual Reconstruction (MARR), which assigns a module-specific scaling coefficient to the residual term of each module. By directly scaling the residual, this coefficient explicitly controls how much cross-layer information is injected into each module, balancing accumulated-error correction against HA bias. In terms of coefficient estimation efficiency, we formulate residual-strength control as a closed-loop feedback problem, where module-level reconstruction error serves as the observable feedback signal for a Proportional-Integral-Derivative (PID)-based adaptive update, avoiding exhaustive coefficient search. Figure 1 provides a conceptual comparison among basic reconstruction, residual reconstruction, and MARR, together with representative performance improvements under 2-bit weight and 4-bit activation quantization (W2A4). (a) Conceptual illustration of our proposed MARR (b) Perplexity comparison under W2A4 on WikiText2 [ 27 ] Figure 1 : Illustration of the motivation and effect of the proposed module-adaptive residual reconstruction. (a) Residual reconstruction improves GPTQ by adding global residual information; however, the residual term may also introduce the bias of Hessian-approximation (HA), leading to suboptimal quantization performance. Instead, MARR controls the residual contribution to balance the benefits and costs of residual. (b) Under W2A4 quantization, MARR consistently improves GPTAQ on WikiText2 [ 27 ] , reducing perplexity by 9.82% on Llama3-8b and 14.94% on Llama2-7b. Our contributions are summarized as follows: • We propose Module-Adaptive Residual Reconstruction (MARR), a new framework to address the suboptimal performance of residual reconstruction-based methods under low-bit PTQ. • We introduce a module-specific scaling coefficient to balance the two-sided effect of residuals, and design a PID-based adaptive update strategy to efficiently estimate the coefficient using reconstruction-error feedback. • Extensive experiments on LLMs and ViTs demonstrate that compared with residual reconstruction baselines, MARR achieves consistent improvements under low-bit quantization. 2 Related Work Low-Bit Post-Training Quantization. Post-training quantization compresses pretrained models into low-precision representations without additional retraining, making it practical for large-scale models [ 22 ] . Existing PTQ methods improve low-bit robustness from several perspectives. Quantization-parameter-based methods [ 47 ] optimize clipping ranges, scaling factors, zero-points, or group-wise configurations to reduce discretization error. Redistribution-based methods reshape weight or activation distributions through smoothing [ 40 ] , re-parameterization [ 20 ] , rotation [ 3 ] , or outlier migration [ 16 ] , making models more amenable to low-bit quantization. Reconstruction-based methods [ 11 , 19 , 17 ] reduce quantization-induced reconstruction error through lightweight weight adjustment, while retaining efficiency via closed-form updates. In this work, we focus on reconstruction-based PTQ and investigate efficient residual reconstruction under low-bit quantization. Reconstruction-Based PTQ. In the reconstruction-based PTQ, for a linear module we adopt the convention that an unadorned symbol denotes the FP quantity and its hat-marked counterpart denotes the quantized one. Specifically, w and \hat{w} denote the FP weight vector and its Quant version, X and \hat{X} denote the FP input (calibration reference) and the corresponding Quant

Read the complete text