Efficiently Training A Flat Neural Network Before It has been Quantizated

Peng Xia, Junbiao PangJiaxin Deng · 2025

Read Efficiently Training A Flat Neural Network Before It has been Quantizated by Peng Xia, Junbiao PangJiaxin Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01462.txt.

Introduction

Peng Xia, Junbiao Pang, and Tianyang Cai prepare neural networks in advance for aggressive quantization, the compression step that shrinks a trained model to a few bits per weight for cheap deployment. Post training quantization of vision transformers is efficient, but ignoring how the full precision model relates to its quantized twin leaves large errors.

Their starting discovery is that a flat full precision network, one sitting in a wide, shallow minimum, quantizes far better. They model activation and weight quantization errors as independent Gaussian noises, disentangle the error sources, and study noise injection training methods that steer the network toward such flat minima. Experiments support the approach, and the authors flag the work as ongoing, with more results to come.

Catalog record

CREATORPeng Xia, Junbiao PangJiaxin Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficiently Training A Flat Neural Network Before It has been Quantizated Peng Xia, Junbiao PangJiaxin Deng Abstract Post-training quantization (PTQ) for vision transformers (ViTs) has garnered significant attention due to its efficiency in compressing models. However, existing methods typically overlook the relationship between a well-trained NN and the quantized model, leading to considerable quantization error for PTQ. However, it is unclear how to efficiently train a model-agnostic neural network which is tailored for a predefined precision low-bit model. In this paper, we firstly discover that a flat full precision neural network is crucial for low-bit quantization. To achieve this, we propose a framework that proactively pre-conditions the model by measuring and disentangling the error sources. Specifically, both the Activation Quantization Error (AQE) and the Weight Quantization Error (WQE) are statistically modeled as independent Gaussian noises. We study several noise injection optimization methods to obtain a flat minimum. Experimental results attest to the effectiveness of our approach. These results open novel pathways for obtaining low-bit PTQ models. 1 Introduction Model compression has become an essential requirement for integrating deep models into edge computing devices. The prevalent methods in the domain of model compression include the search for optimal neural architectures Zoph and Le 2016 , network pruning Han et al. 2015 , and the Deep Neural Network (DNN) quantization Li et al. 2021a Esser et al. 2019 . DNN quantization are categorized into two sub-classes: Post-Training Quantization (PTQ) Nagel et al. 2020 , Li et al. 2021a , Wei et al. 2022 , Li et al. 2023b and Quantization-Aware Training (QAT) Esser et al. 2019 , Nagel et al. 2022 . PTQ adjusts the quantized model with a limited calibration dataset, bypassing the need for retraining. However, when dealing with low-bit widths, e.g. , 2 or 4 bits, PTQ may face a significant drop in performance. Despite the demonstrated success of these advanced PTQ methods—for instance, QDrop Wei et al. 2022 optimizes quantization policies by simulating information loss, while SmoothQuant Xiao et al. 2024 mitigates activation outliers in large models by transferring the quantization difficulty onto weights—we contend that they all operate under a fundamental constraint: their ultimate performance is capped by the intrinsic properties of the pre-trained, full-precision model. Specifically, the central tenet of state-of-the-art PTQ approaches is to minimize a form of reconstruction error. This error quantifies the deviation of the quantized model’s output from that of its full-precision counterpart, which is invariably treated as the ground truth. The objective of PTQ is, therefore, to emulate the functionality of the original model within the confines of a discrete parameter space. This framing, however, raises a critical and often-overlooked question: if the full-precision model itself is inherently sensitive to parameter perturbations, can any emulation of it be robust? Empirical investigations reveal that high-performing full-precision models derived from standard training often converge to sharp loss landscapes. As illustrated in Figure 1 , these models exhibit acute sensitivity to minor parameter perturbations introduced by quantization. Even state-of-the-art PTQ algorithms merely attempt to ameliorate an inherently quantization-averse model, akin to balancing on a needle’s point: regardless of the sophistication of the balancing technique, the intrinsic instability renders the endeavor precarious. This instability arises from the deterministic perturbations imposed by quantization on model weights \boldsymbol{W} and activations \boldsymbol{x} , denoted as \Delta\boldsymbol{W} and \Delta\boldsymbol{x} . For a given input \boldsymbol{x} , the quantized model’s loss \boldsymbol{L_{Q}} can be approximated via Taylor expansion of the full-precision loss \boldsymbol{L_{FP}} : \begin{split}\boldsymbol{L_{Q}}&=\boldsymbol{L}(\boldsymbol{W}+\Delta\boldsymbol{W},\boldsymbol{x}+\Delta\boldsymbol{x})\\ &\approx\boldsymbol{L}(\boldsymbol{W},\boldsymbol{x})+(\nabla\boldsymbol{L})^{T}\cdot[\Delta\boldsymbol{W};\Delta\boldsymbol{x}]\\ &\quad+0.5\cdot[\Delta\boldsymbol{W};\Delta\boldsymbol{x}]^{T}\cdot\boldsymbol{H}\cdot[\Delta\boldsymbol{W};\Delta\boldsymbol{x}]\end{split} (1) where \boldsymbol{H} denotes the Hessian matrix. This formulation elucidates that the loss degradation, \boldsymbol{\Delta L}=\boldsymbol{L_{Q}}-\boldsymbol{L_{FP}} , is predominantly governed by the second-order term, as the gradient \nabla\boldsymbol{L} approaches zero at a loss minimum. Consequently, the norm of the Hessian \|\boldsymbol{H}\| , which quantifies the curvature or sharpness of the loss landscape, directly modulates the model’s sensitivity to perturbations. A flat landscape (small \|\boldsymbol{H}\| ) implies that quantization-induced perturbations [\Delta\boldsymbol{W};\Delta\boldsymbol{x}] yield minimal \boldsymbol{\Delta L} . Prevailing PTQ methods strive to minimize the perturbations \Delta\boldsymbol{W} and \Delta\boldsymbol{x} themselves through optimized quantization schemes; however, they overlook the potential to minimize \|\boldsymbol{H}\| via tailored training processes. In summary, contemporary PTQ paradigms lack a mechanism to proactively steer full-precision models toward minima that are inherently robust to quantization perturbations, i.e., flat minima. Building upon this analysis, we introduce a novel paradigm termed Differential Noise-driven Quantization-aware Training (DNQ) , which implicitly regularizes the Hessian matrix to induce a flat loss landscape, thereby facilitating subsequent PTQ. Intuitively, our approach eschews passive acceptance of a pretrained model; instead, it exposes the model to simulated quantization noise during training, compelling the optimizer to seek solutions that perform robustly not only at the current point but across its neighborhood, thereby naturally converging to flat minima. We explore this objective both theoretically and empirically, with the following key contributions: • Theoretically, we reframe the challenge of achieving high-performance PTQ as optimizing the flatness of the full-precision model’s loss landscape. We mathematically demonstrate the direct correlation between quantization-induced performance degradation and the Hessian norm, providing a rigorous foundation for training quantization-friendly models. • Methodologically, we propose the DNQ framework to implicitly minimize the Hessian norm. This framework periodically measures and models Weight Quantization Error (WQE) and Activation Quantization Error (AQE) via simulated PTQ. We innovatively employ differential noise injection for stable weight training, coupled with a two-stage strategy to balance convergence and robustness. • Empirically, our solution yields substantial advancements across multiple benchmark datasets and network architectures. Full-precision models trained with our solution, when subjected to simple PTQ, outperform those optimized with complex PTQ algorithms on standard pretrained baselines, validating the efficacy of our approach. 2 Related Work 2.1 Post-Training Quantization Model quantization Krishnamoorthi 2018 is primarily pursued through Quantization-Aware Training (QAT) Esser et al. 2019 , Nagel et al. 2022 , Huang et al. 2024 and Post-Training Quantization (PTQ) Wei et al. 2022 , Ma et al. 2023 , Li et al. 2023a , Frantar et al. 2023 . While QAT achieves high accuracy via resource-intensive retraining, PTQ has emerged as a lightweight alternative, converting a pre-trained model directly. The PTQ methodology has evolved from minimizing layer-wise weight error to a data-driven, reconstruction-based paradigm. This modern approach, which underpins most SOTA methods, minimizes the feature map error between the FP and quantized models. This line of work was pioneered by AdaRound Nagel et al. 2020 , which used second-order information (Hessian) to guide layer-wise rounding. BRECQ Li et al. 2021a subsequently advanced this by extending the optimization granularity from layer-wise to block-wise, using the Fisher Information Matrix as a proxy for the Hessian to better compensate for inter-layer errors. This block-wise strategy is now a cornerstone in complex tasks, such as diffusion model quantization Li et al. 2023a , Sui et al. 2024 . Further refinements have focused on robustness. QDrop Wei et al. 2022 links loss landscape flatness to generalization, introducing dropout during reconstruction to guide optimization towards flatter minima. MRECG Ma et al. 2023 analyzes and mitigates error accumulation by adapting the reconstruction granularity. Despite their sophistication, these PTQ methods are fundamentally post-hoc. They all operate on a given, pre-trained full-precision model. As we argued in Section 1 , if this initial model resides in a sharp, quantization-sensitive loss minimum, the

Read the complete text