Theory-optimal Quantization Based on Flatness

Xiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu · 2026

Read Theory-optimal Quantization Based on Flatness by Xiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18800.txt.

Introduction

Xiusheng Huang and seven colleagues tackle outlier damage in post-training quantization of large language models. Outliers hurt accuracy most at low precision, and earlier transforms still leave concentrated magnitude patterns. The authors model how quantization error links to outliers and define Flatness to measure outlier spread. Their Bidirectional Diagonal Quantization spreads outlier magnitudes through tuned matrix transforms. Tests show it disperses the patterns that earlier methods leave behind.

Catalog record

CREATORXiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51008
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Theory-optimal Quantization Based on Flatness Xiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu Abstract Post-training quantization has emerged as a widely adopted technique for compressing and accelerating the inference of Large Language Models (LLMs). The primary challenges in LLMs quantization stem from activation outliers, which significantly degrade model performance especially at lower bit precision. While recent approaches attempt to mitigate outliers through linear transformations across feature dimensions, our analysis reveals that the transformed weights and activations still exhibit persistent outlier patterns with concentrated magnitude distributions. In this paper, we first model the mathematical relationship between quantization error and outliers, and then introduce a new metric Flatness to quantify the distribution of outliers. Based on this, we derive the theoretical optimal solution with respect to Flatness. Building on these insights, we propose Bidirectional Diagonal Quantization (BDQ), a novel post-training quantization framework that effectively disperses outlier patterns through optimized matrix transformations. BDQ strategically distributes outlier magnitudes across matrix dimensions via learned diagonal operations. Extensive experiments demonstrate that BDQ establishes a new quantization benchmark. It achieves less than 1% accuracy drop in W4A4 quantization on the LLaMA-3-8B model. In the more challenging W2A4KV16 experiment, compared to state-of-the-art approaches, BDQ reduces the performance gap by 39.1% on the DeepSeek-R1-Distill-LLaMA-70B model. † † footnotetext: \dagger Corresponding authors 1 Introduction Recent Large Language Models (LLMs) have achieved superior performance in multiple natural language processing tasks as their parameters grow ( 32 ; 9 ) . However, increasing the scale of the parameters leads to significant increases in computational and storage costs ( 31 ) . Therefore, the efficient deployment of low-cost LLMs has become an urgent research direction ( 1 ) . Previous research can be divided into architecture-changing and architecture-preserving techniques. Architecture-changing methods such as distillation ( 11 ; 5 ) and pruning ( 38 ) reduce the size of the model by transferring knowledge or removing unimportant parameters, but require significant data and computation, making them impractical for LLMs. In contrast, architecture-preserving methods such as quantization ( 6 ) and low-rank decomposition ( 34 ) keep the model structure; quantization lowers weight precision, while low-rank methods approximate weight matrices. Quantization is especially popular in LLM deployment due to its efficiency and strong performance. Post-Training Quantization (PTQ) has become a widely adopted technique for compressing and accelerating LLMs. During quantization, as shown in Figure 1(a) , outliers in the original data present huge challenges because the limited quantization space cannot adequately express the original data space, with most data accumulating in a few regions. Recent research has adopted linear transformations to address these challenges. The rotation transformation ( 1 ; 19 ) alleviates this phenomenon in Figure 1(b) . However, due to the presence of outliers, most of the data still accumulates in the Blue region. Existing methods are heuristic and haven’t established direct mathematical relationships between outliers and quantization errors, nor optimized the distribution of the entire quantization space. (a) Original (b) Rotation Transformation (c) Ours Fig. 1: Activation distributions under different transformations for LLaMA3-8B. After quantization, values from various ranges are mapped to corresponding integer levels. The number of points within each box reflects the frequency of quantized values. A more uniform distribution of points indicates higher quantization quality. Blue dots represent values near zero, Orange dots indicate mid-range values, and Red dots correspond to large-magnitude values. In this paper, we first establish the mathematical relationship between outliers and quantization errors, demonstrating that outliers influence quantization error at the quadratic level. Furthermore, we introduce the concept of Flatness as an effective indicator for quantifying the distribution of outliers. Inspired by Information-Entropy ( 29 ) , we define Flatness as evaluating each element’s flatness in its row and column, extending to all elements in the matrix. Importantly, through mathematical derivation, we discovered the optimal solution for improving Flatness and demonstrated excellent advantages compared to previous methods, laying the foundation for developing more effective quantization methods. Based on the above findings, we propose the B idirectional D iagonal Q uantization ( BDQ ) method. BDQ allocates two learnable diagonal transformation pairs for each fully connected layer in LLMs, applying simultaneous row-wise and column-wise scaling to redistribute outliers along both dimensions. We theoretically demonstrate that this formulation can achieve the optimal solution with respect to Flatness. In addition, a Hadamard orthogonal transformation is employed to further disperse outliers across the entire matrix. Meanwhile, it is widely known that only a small calibration set is utilized (e.g., 128 samples) during the quantization process, which can cause the model to overfit to a limited set of features, an effect shown experimentally to hinder outlier mitigation. To address this, BDQ introduces a Recursive Cross-Entropy loss that captures the state from previous iterations, thereby reducing overfitting and improving generalization. BDQ is a highly effective PTQ method for LLMs, consistently outperforming existing techniques across various models and benchmarks. In the W4A4KV4 setting, BDQ maintains over 99.1% of full-precision accuracy. Furthermore, in the W2A4KV16 setting, BDQ reduces the performance gap of DeepSeek-R1-Distill-LLaMA-70B ( 10 ) by 39.1% compared to the latest methods. To our knowledge, we are the first to model the mathematical relationship between outliers and quantization errors, discovering that outliers are key factors affecting quantization accuracy. Meanwhile, we propose the Flatness metric reflecting the presence of outliers in the model and provide the optimal solution through mathematical derivation. The contributions of this work are summarized as follows: • We first model the mathematical relationship between outliers and quantization errors, discovering that outliers are key factors that affect quantization accuracy. • To quantify the outlier distribution, we propose the Flatness metric and provide the optimal solution through mathematical derivation. Compared to previous methods, this optimal solution demonstrates significant advantages. • We propose a Bidirectional Diagonal Quantization (BDQ) that effectively reduces quantization errors. Extensive experiments show that BDQ significantly outperforms existing quantization methods. 2 Related Work 2.1 Architecture-Changing Methods Recent model compression research has focused on structural modifications to reduce complexity and size. Pruning techniques have progressed from early weight pruning ( 11 ) to dynamic strategies that remove unimportant parameters during training ( 5 ) , and to neural architecture search-based methods for optimal network structures ( 36 ) . Knowledge distillation has also advanced, from foundational teacher-student frameworks ( 15 ) to approaches combining self-supervised learning ( 33 ) and multi-modal distillation for preserving semantics ( 37 ) . However, these methods often entail high computational costs and slow processing, limiting their practical deployment. 2.2 Architecture-Preserving Methods Post-training quantization (PTQ) is popular in LLMs for its efficiency, with methods mainly divided into weight-only and weight-activation quantization. FWSVD ( 12 ) and ASVD ( 34 ) assess parameter or channel importance, while GPTQ ( 6 ) and AWQ ( 17 ; 16 ) reduce quantization error and address activation outliers. QuIP ( 4 ) , QuIP# ( 30 ) , SmoothQuant ( 31 ) , and OmniQuant ( 25 ) further improve quantization with various techniques. I-LLM ( 14 ) supports integer-only inference, QuaRot ( 1 ) uses random rotations, and SpinQuant learns rotations for 4-bit quantization ( 19 ) . Quantization stands out over low-rank decomposition for its high accuracy and low

Read the complete text