HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning

Jinhao Zhang, Yunquan Zhang, Zicheng Yan, Boyang Zhang, Jun Sun, Daning Cheng · 2026

Read HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning by Jinhao Zhang, Yunquan Zhang, Zicheng Yan, Boyang Zhang, Jun Sun, Daning Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21626.txt.

Introduction

Jinhao Zhang, Yunquan Zhang, Zicheng Yan, Boyang Zhang, Jun Sun, and Daning Cheng fix a paradox in compressing language models. Post-training quantization shrinks models by rounding their numbers. It often shows low rounding error yet high loss, which seems contradictory.

The cause lies in the shape of the loss landscape. A few high-curvature directions are extremely sensitive to small nudges. Errors there cost far more than errors elsewhere. Their method conditions the problem using that curvature map, so rounding avoids the fragile directions. Steadier compression keeps small models accurate enough for real use on modest hardware.

Catalog record

CREATORJinhao Zhang, Yunquan Zhang, Zicheng Yan, Boyang Zhang, Jun Sun, Daning Cheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning Jinhao Zhang, Yunquan Zhang, Zicheng Yan, Boyang Zhang, Jun Sun, Daning Cheng Abstract Post-Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical “low-error, high-loss” phenomenon because it focuses solely on minimizing quantization error. The root cause lies in the Hessian matrix of the LLM loss landscape: a few high-curvature directions are extremely sensitive to perturbations. To address this, we propose the Hessian Robust Quantization (HeRo-Q) algorithm, which applies a lightweight, learnable rotation-compression matrix to the weight space prior to quantization. This joint framework reshapes the loss landscape by reducing the largest Hessian eigenvalue and reducing its max eigenvalue, thereby significantly enhancing robustness to quantization noise. HeRo-Q requires no architectural modifications, incurs negligible computational overhead, and integrates seamlessly into existing PTQ pipelines. Experiments on Llama and Qwen models show that HeRo-Q consistently outperforms state-of-the-art methods—including GPTQ, AWQ, and SpinQuant—not only achieving superior performance under standard W4A8 settings, but also excelling in the highly challenging W3A16 ultra-low-bit regime, where it boosts GSM8K accuracy on Llama-3-8B to 70.15% and effectively avoids the logical collapse commonly seen in aggressive quantization. 1 Introduction (a) low-error, high-loss for some weight parameters (b) local geometry of the loss landscape Figure 1 : a: Certain parameters suffer severe performance degradation even under small quantization noise because their noise components project significantly onto the high-curvature (short-axis) directions. Noise amplitude is fixed across all trials in the figure. b: The local geometry of the loss landscape around a converged point can be approximated by a hyper-ellipsoid: its short axis corresponds to the direction of the largest Hessian eigenvalue (high curvature). In contrast, its long axis aligns with the smallest eigenvalue (low curvature). A small perturbation along the short axis causes a dramatic increase in loss, whereas a perturbation of the same magnitude along the long axis has a negligible effect. Post-Training Quantization (PTQ) is a key technology nowadays. Current mainstream PTQ methods primarily aim to minimize quantization error. They typically employ symmetric or asymmetric linear mapping schemes, use a calibration dataset to estimate dynamic ranges, and select optimal quantization parameters by minimizing statistical discrepancies between pre-quantized and post-quantized weights or activations. Although these strategies work well for traditional models like those in computer vision, they often encounter a “low-error, high-loss” paradox when applied to modern LLMs: even with extremely low overall quantization error, downstream task performance can still degrade substantially, as shown in Figure 1(a) . This suggests the existence of highly sensitive parameters that, despite contributing little to overall noise, exert a disproportionately large impact on model behavior. The root cause of this degradation lies in the geometric structure of the LLM loss landscape—particularly the ill-conditioned nature of its Hessian matrix. Specifically, the Hessian spectrum of LLMs is highly non-uniform: a few directions exhibit very large eigenvalues (i.e., high curvature), while most directions have near-zero eigenvalues (i.e., flat regions). This strong anisotropy makes the model exquisitely sensitive to tiny perturbations along high-curvature directions. Since quantization noise is injected uniformly across all dimensions, even minimal total noise can cause significant performance collapse if it has any non-zero component along these sensitive directions, as shown in Figure 1(b) . Therefore, the key to enhancing the stability of model quantization lies in reducing the large eigenvalues of the Hessian matrix through low-cost transformations. In this paper, we propose the Hessian Robust Quantization (HeRo-Q) algorithm, a novel quantization framework. Instead of directly minimizing quantization error, HeRo-Q reconstructs the weight space via a synergistic joint transformation strategy. Specifically, the diagonal smoothing component actively compresses the Hessian’s spectral radius to tame the outlier curvature, while the learnable orthogonal rotation redistributes quantization noise across dimensions to minimize its alignment with high-sensitivity directions. By decoupling these geometric roles, HeRo-Q effectively lowers the model’s worst-case sensitivity to quantization perturbations. Importantly, HeRo-Q requires no changes to model architecture or activation functions—it introduces only a lightweight, learnable linear transformation on weights, making it fully compatible with existing PTQ pipelines for fast and effective quantization, shown in Figure 2 . Figure 2 : HeRo-Q stabilizes quantization by reshaping the Hessian spectrum. Through a lightweight linear transformation, HeRo-Q attenuates large eigenvalues—particularly outliers far from the bulk of the spectrum—and caps the maximum eigenvalue, thereby reducing the loss landscape’s sensitivity to quantization noise and enabling robust, efficient low-bit inference. Our contributions are as follows: (1) We establish a theoretical connection between quantization robustness and Hessian spectral properties from the perspective of loss landscape geometry, and propose the HeRo-Q algorithm accordingly; (2) HeRo-Q applies only a single learnable matrix multiplication before quantization, incurring negligible extra computational cost; (3) In experiments, HeRo-Q consistently outperforms state-of-the-art PTQ methods (e.g., GPTQ, AWQ, SpinQuant) on both Llama and Qwen model families. It achieves superior language modeling and reasoning accuracy under different quantization configurations with low inference cost. 2 Related Work Contemporary quantization strategies diverge sharply in how they allocate numerical precision across neural networks. On one end of the spectrum lie uniform-bitwidth approaches, which—despite their simplicity—impose a single precision level globally (see, e.g., ( 18 ; 19 ; 17 ; 20 ; 22 ; 21 ) ). While easy to implement, they often sacrifice performance for consistency. In response, mixed-precision schemes emerged, dynamically tailoring bit-widths per layer to strike a nuanced compromise between computational thrift and predictive fidelity. Search-driven quantization frames bit-width assignment as a combinatorial optimization problem, with methods like AutoQ ( 8 ) , HAQ ( 12 ) , and mixed-precision schemes ( 13 ) iteratively exploring precision configurations guided by accuracy or loss to find Pareto-optimal solutions—yet their exhaustive search is computationally prohibitive for large models. Sensitivity-aware quantization offers a more efficient alternative by compressing layers according to their error tolerance: robust layers use low precision, while sensitive ones retain higher bit-widths. Layer sensitivity is estimated via metrics such as KL divergence ( 1 ; 15 ) , Hessian-based curvature ( 4 ; 16 ) , Gauss–Newton approximations ( 3 ) , or custom scores ( 11 ) , all aiming to minimize total network sensitivity under a fixed compression budget. Figure 3 : Robust Quantization via Hessian Transformation: HeRo-Q first analyzes the Hessian structure and identifies a severe Hessian geometric skewness —certain directions exhibit extremely high curvature (corresponding to large eigenvalues) and are highly sensitive to quantization noise. To address this, HeRo-Q searches over smoothing parameters \alpha to construct a diagonal scaling matrix \mathbf{D}_{\alpha} , and learns an orthogonal rotation matrix \mathbf{R} via the Cayley transform. Together, they form a linear transformation \mathbf{T}=\mathbf{D}_{\alpha}^{-1}\mathbf{R} that maps the original weight space into a geometrically more balanced transformed space. Robust quantization is then performed in this transformed space, and the quantized weights in the original space are recovered through the inverse transformation. Beneath these algorithmic innovations lies a bedrock of foundational techniques that reshape post-training quantization. AdaRound ( 9 ) rethinks rounding as a learnable reconstruction task; GPTQ ( 5 ) and AWQ ( 6 ) demonstrate that preserving salient weights or activating outlier-aware scaling can rescue ultra-low-bit performance; SmoothQuant ( 14 ) tames activation outliers through input-aware rescaling. The latest wave pushes further: QuIP ( 2 ) injects incoherence to stabilize extreme quantization; OmniQuant ( 10 ) makes clipping thresholds

Read the complete text