Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization
YiFeng Wang, Zhun Sun, Keisuke Sakaguchi · 2026
Read Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization by YiFeng Wang, Zhun Sun, Keisuke Sakaguchi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00140.txt.
Introduction
YiFeng Wang, Zhun Sun, and Keisuke Sakaguchi present a report on Activation Residual Hessian Quantization, a method for compressing large language models to very few bits. Low-bit quantization of both activations and weights lets errors propagate through a network. Their method is a post-training weight splitting scheme that identifies where those errors hurt most and protects those directions.
From the residuals of activation rounding they build an input-side residual Hessian, then use a closed-form truncated singular value split to isolate error-sensitive weight directions into a high-precision low-rank branch. Experiments on a recent Qwen thinking model show clearly better layer-wise signal quality and preserved reasoning results on a logic benchmark even under aggressive quantization. The code is publicly available.
Catalog record
| CREATOR | YiFeng Wang, Zhun Sun, Keisuke Sakaguchi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization YiFeng Wang, Zhun Sun, Keisuke Sakaguchi Abstract We present Activation Residual Hessian Quantization (ARHQ), a post-training weight splitting method designed to mitigate error propagation in low-bit activation-weight quantization. By constructing an input-side residual Hessian from activation quantization residuals ( G x G_{x} ), ARHQ analytically identifies and isolates error-sensitive weight directions into a high-precision low-rank branch. This is achieved via a closed-form truncated SVD on the scaled weight matrix W G x 1 / 2 WG_{x}^{1/2} . Experimental results on Qwen3-4B-Thinking-2507 demonstrate that ARHQ significantly improves layer-wise SNR and preserves downstream reasoning performance on ZebraLogic even under aggressive quantization. The code is available at https://github.com/BeautMoonQ/ARHQ . 1 Introduction Low-bit quantization of Large Language Models (LLMs) frequently encounters a fundamental mismatch between compression efficiency and inference fidelity. Conventional low-rank splitting methods typically approach this by addressing different sources of information loss: (1) minimizing the Frobenius energy to reconstruct the weight matrix W ( 3 ) ; (2) preserving heavy-hitter features by weighting the objective with activation energy H_{x}=\frac{1}{N}X^{\top}X ( 4 ) ; or (3) absorbing magnitude outliers into a high-precision branch to reduce the dynamic range and clipping errors of the quantized main branch ( 2 ) . However, we observe that under aggressive activation quantization, the dominant source of output error is often not the loss of original signal energy nor the presence of magnitude outliers alone, but rather how the linear map amplifies quantization residuals originating from the input activations. This observation motivates Activation Residual Hessian Quantization (ARHQ). Unlike previous works that aim to protect the signal or absorb the outliers , ARHQ is designed specifically to suppress the noise amplification . By decomposing the output error of a quantized linear layer, we isolate the specific term E_{x}W^{\top} responsible for propagating activation residuals E_{x} . ARHQ optimizes a low-rank split for this term, re-casting the task as a weighted low-rank approximation problem under a metric induced by the residual covariance G_{x}=\frac{1}{N}E_{x}^{\top}E_{x} . Intuitively, ARHQ identifies weight directions that most strongly amplify quantization noise and assigns them to a high-precision low-rank branch, while the remaining ”residual-insensitive” components follow the standard quantized path. The optimization objective of ARHQ represents a significant departure from existing paradigms: Weight-only Reconstruction ( 3 ) : Minimizes pure weight approximation error by \min_{L}\lVert W-L\rVert_{F}^{2}; (1) Outlier Absorption ( 2 ) : Extracts magnitude outliers into L to minimize the quantization clipping error of the residual main branch by \min_{L}\lVert(W-L)-Q_{w}(W-L)\rVert_{F}^{2}; (2) Activation-aware Reconstruction ( 4 ) : Weights the reconstruction by the input activation covariance H_{x} by \min_{L}\lVert X(W-L)^{\top}\rVert_{F}^{2}\approx\min_{L}\lVert(W-L)H_{x}^{1/2}\rVert_{F}^{2}; (3) Activation-Residual-aware Reconstruction : Weights the reconstruction by the quantization residual covariance G_{x} by \min_{L}\lVert E_{x}(W-L)^{\top}\rVert_{F}^{2}=\min_{L}\lVert(W-L)G_{x}^{1/2}\rVert_{F}^{2}. (4) We justify the term Residual Hessian by demonstrating that G_{x} is precisely the shared input-side Hessian block of the residual propagation loss. This distinction is critical: since G_{x} is derived directly from the observed quantization error E_{x} , ARHQ is inherently quantizer-adaptive. It adapts to the specificities of the deployed hardware codebooks, scaling rules, and calibration distributions, rather than assuming isotropic or quantizer-agnostic noise. 2 Activation Residual Hessian Quantization 2.1 Quantization Error Decomposition Consider a linear layer defined by Y=XW^{\top}, (5) where X\in\mathbb{R}^{N\times D_{\mathrm{in}}} is the input activation matrix and W\in\mathbb{R}^{D_{\mathrm{out}}\times D_{\mathrm{in}}} is the weight matrix. Let Q_{x}(\cdot) and Q_{w}(\cdot) denote the activation and weight quantizers, respectively. We maintain a consistent notation for quantization residuals throughout this report: E_{x}=Q_{x}(X)-X,\qquad E_{w}=Q_{w}(W)-W (6) The actual forward pass of the quantized layer is given by \hat{Y}=Q_{x}(X)Q_{w}(W)^{\top}=(X+E_{x})(W+E_{w})^{\top} (7) Expanding this expression reveals the full output error \hat{Y}-Y=E_{x}W^{\top}+XE_{w}^{\top}+E_{x}E_{w}^{\top}. (8) Equation ( 8 ) separates the total error into three distinct components: activation-error propagation ( E_{x}W^{\top} ), weight-error propagation ( XE_{w}^{\top} ), and their second-order interaction. ARHQ is specifically designed to tackle the first term, E_{x}W^{\top} , as it directly captures how the layer amplifies activation quantization noise, which is considered a dominant source of degradation in low-bit regimes, and conveniently admits a tractable closed-form solution. 2.2 Isolating the Residual Propagation via a Low-Rank Branch To mitigate this error amplification, ARHQ splits the weight matrix into two fundamental components: W=W_{\mathrm{res}}+L,\qquad\mathrm{rank}(L)\leq r (9) Here, W_{\mathrm{res}} remains on the primary quantized branch, while L acts as a high-precision, low-rank side branch. In practical deployments, to satisfy the rank constraint and achieve actual computational and memory savings, this low-rank matrix is explicitly parameterized as the product of two factor matrices as L=BA^{\top} , where A\in\mathbb{R}^{D_{\mathrm{in}}\times r} acts as a down-projection and B\in\mathbb{R}^{D_{\mathrm{out}}\times r} acts as an up-projection (the exact analytical forms of B and A will be derived in Section 2.5). During actual inference, the forward pass of the layer is structurally approximated by these two parallel branches as \hat{Y}=Q_{x}(X)Q_{w}(W_{\mathrm{res}})^{\top}+XL^{\top} (10) To derive a closed-form split for L , we must isolate the activation-residual propagation. We do this by temporarily assuming the main branch W_{\mathrm{res}} is kept in full precision, yielding a surrogate inference model denoted as \hat{Y}_{\mathrm{approx}}=Q_{x}(X)W_{\mathrm{res}}^{\top}+XL^{\top} (11) Substituting W_{\mathrm{res}}=W-L into this surrogate model, we evaluate the gap between the exact floating-point output and the surrogate approximation Y-\hat{Y}_{\mathrm{approx}}=XW^{\top}-Q_{x}(X)(W-L)^{\top}-XL^{\top}=(X-Q_{x}(X))(W-L)^{\top}. (12) Since the activation residual is defined as E_{x}=Q_{x}(X)-X , the error magnitude becomes exactly E_{x}(W-L)^{\top} . Minimizing the Frobenius energy of this isolated error leads directly to the ARHQ primal objective as \min_{\mathrm{rank}(L)\leq r}\lVert E_{x}(W-L)^{\top}\rVert_{F}^{2} (13) 2.3 The Activation Residual Hessian To solve this objective analytically, we define the residual covariance matrix in the following form: G_{x}=\frac{1}{N}E_{x}^{\top}E_{x}. (14) Using the cyclic property of the trace, we can rewrite the primal objective as \displaystyle\lVert E_{x}(W-L)^{\top}\rVert_{F}^{2} \displaystyle=\mathrm{Tr}\!\left((W-L)E_{x}^{\top}E_{x}(W-L)^{\top}\right) (15) \displaystyle=N\,\mathrm{Tr}\!\left((W-L)G_{x}(W-L)^{\top}\right) (16) \displaystyle=N\,\lVert(W-L)G_{x}^{1/2}\rVert_{F}^{2}. (17) Dropping the constant factor N , the optimization problem simplifies to a weighted low-rank approximation that solves \min_{\mathrm{rank}(L)\leq r}\lVert(W-L)G_{x}^{1/2}\rVert_{F}^{2}. (18) It is important to explicitly distinguish the residual covariance G_{x} from the standard activation covariance H_{x}=\frac{1}{N}X^{\top}X . In standard post-training quantization literature ( 1 ) , weight importance is typically governed by the activation Hessian H_{x} , which arises naturally as the second-order derivative of the standard signal reconstruction loss. Methods relying on H_{x} inherently assume that preserving the largest original input features is the optimal strategy. In contrast, ARHQ relies on G_{x} to directly capture the structure of the quantization noise itself. This ensures that the low-rank split prioritizes fixing the errors introduced by activation quantization, rather than generic signal reconstruction. Because our primary focus shifts from full signal reconstruction to mitigating the isolated residual propagation error, the optimization landscape is correspondingly governed by G_{x} . We therefore formally term G_{x} the activation residual Hessian . To see this mathematically, consider the residual propagation loss derived from our isolated objective: \mathcal{L}(W_{\mathrm{res}})=\frac{1}{N}\lVert E_{x}W_{\mathrm{res}}^{\top}\rVert_{F}^{2}=\mathrm{Tr}(W_{\mathrm{res}}G_{x}W_{\mathrm{res}}^{\top}) . Its gradient and Hessian with respect to W_{\mathrm{res}} can be exactly written as: \nabla_{W_{\mathrm{res}}}\mathcal{L}=2W_{\mathrm{res}}G_{x},\qquad\nabla^{2}_{W_{\mathrm{res}}}\mathcal{L}=2I_{D_{\mathrm{out}}}\otimes G_{x}. (19) Thus, G_{x} plays the exact same mathematical and functional role in our residual-aware framework as the standard Hessian H_{x} does in traditional methods like GPTQ ( 1 ) . 2.4 Closed-form solution Proposition 1 (ARHQ closed-form decomposition) . Let W\in\mathbb{R}^{D_{\mathrm{out}}\times D_{\mathrm{in}}} and assume the activation residual covariance G_{x}\succ 0 . Define the scaled weight matrix M=WG_{x}^{1/2} . If M_{r} is the optimal rank- r approximation of M under the Frobenius norm, then the exact solution to the weighted objective \min_{\mathrm{rank}(L)\leq r}\lVert(W-L)G_{x}^{1/2}\rVert_{F}^{2} (20) is strictly given by L^{\star}=M_{r}G_{x}^{-1/2} (21)