Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantization

Stone Yun, Alexander Wong · 2025

Read Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantization by Stone Yun, Alexander Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10463.txt.

Introduction

Stone Yun and Alexander Wong ask whether the starting weights of a neural network decide how well it survives quantization, the shrinking of numbers that makes models cheap to run. Most quantization research improves training recipes or adds penalties, and leaves the initialization to habit. The authors ran a broad study across the building blocks of efficient image networks, swapping in different random starting schemes.

The starting scheme, they found, can shift final quantized accuracy a lot, whatever the architecture. They then tried a different source of starting weights: a graph hypernetwork that predicts a network's parameters from its structure. Networks started from those predictions proved unusually sturdy under quantization. Fine-tuning the predictor for quantized graphs pushed accuracy higher still, with clear gains even at four bits and better-than-random results at two bits. The authors present initialization as an overlooked lever in efficient model design.

Catalog record

CREATORStone Yun, Alexander Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantization Stone Yun, Alexander Wong Abstract Deep neural network (DNN) quantization for fast, efficient inference has been an important tool in limiting the cost of machine learning (ML) model inference. Quantization-specific model development techniques such as regularization, quantization-aware training, and quantization-robustness penalties have served to greatly boost the accuracy and robustness of modern DNNs. However, very little exploration has been done on improving the initial conditions of DNN training for quantization. Just as random weight initialization has been shown to significantly impact test accuracy of floating point models, it would make sense that different weight initialization methods impact quantization robustness of trained models. We present an extensive study examining the effects of different weight initializations on a variety of CNN building blocks commonly used in efficient CNNs. This analysis reveals that even with varying CNN architectures, the choice of random weight initializer can significantly affect final quantization robustness. Next, we explore a new method for quantization-robust CNN initialization — using Graph Hypernetworks (GHN) to predict parameters of quantized DNNs. Besides showing that GHN-predicted parameters are quantization-robust after regular float32 pretraining (of the GHN), we find that finetuning GHNs to predict parameters for quantized graphs (which we call GHN-QAT) can further improve quantized accuracy of CNNs. Notably, GHN-QAT shows significant accuracy improvements for even 4-bit quantization and better-than-random accuracy for 2-bits. To the best of our knowledge, this is the first in-depth study on quantization-aware DNN weight initialization. GHN-QAT offers a novel approach to quantized DNN model design. Future investigations, such as using GHN-QAT-initialized parameters for quantization-aware training, can further streamline the DNN quantization process. 1 Introduction Besides exponentially increasing modern massive models, there are many use-cases with a need for resource efficient deep neural networks (DNN). For example, embedded systems and other mobile AI computing requires fast, efficient DNNs that fit tight budgets on computation, latency, and power. In terms of efficient inference, developing DNNs for resource constrained AI has led to several different approaches to tackle the efficiency-accuracy tradeoff. These include fast, computationally efficient DNN architectures [ 1 , 2 , 3 ] and limited precision quantization for low-power inference [ 4 , 5 , 6 ] . In resource constrained AI, combining computationally efficient architectures with fixed-precision, integer quantization such as in [ 5 ] has become an essential tool for deploying fast, efficient convolutional neural networks (CNN) to the edge. However, perturbations induced by quantization of weights and activations can change DNN behaviour in non-trivial ways and developing efficient models for low-power, quantized inference while retaining close to original floating point accuracy is a very challenging task. In some cases, state-of-the-art performance can have significant degradation after quantization of weights and activations [ 7 ] . Despite significant works demonstrating ways to recover near floating-point performance using reduced precision inference, there is still limited understanding of how different design decisions can affect the quantized inference behaviour of CNNs. For example, random weights initialization strategies are often designed with the goal of solving issues such as vanishing/exploding gradient [ 8 , 9 , 10 ] . However, an often-overlooked aspect of weights initialization is its impact on the final trained distributions of each layer. As they determine our starting point on the loss surface, initial distributions of each weight tensor will significantly impact the final trained distributions of a model. With regards to quantization, this means that weight initialization choices will impact the dynamic ranges of the weights and activations in a trained CNN as well as other aspects of the learned distributions. Thus, affecting the noise in our system and the expected quantized inference behaviour. Besides randomly initialized neural networks, Graph Hypernetworks (GHN) have also become an exciting new method for initializing DNNs with higher accuracy. Recent works by [ 11 ] and [ 12 ] have shown remarkable accuracy using GHNs to predict all trainable parameters of unseen DNNs in a single forward pass . For example, [ 11 ] report that their GHN-2 can predict the parameters of an unseen ResNet-50 to achieve 60% accuracy on CIFAR-10. Thus, saving thousands of training steps compared to random initialization plus iterative optimization. We adapt GHNs for quantization in our investigation and further explore how GHNs can be used for better initialization of quantized CNNs. While GHN-predicted parameters are still significantly less accurate than iteratively optimized models, the high accuracy of GHN-predicted parameters could potentially make for an excellent parameter initialization that significantly amortizes a couple thousand steps of training. Previous works [ 11 , 12 , 13 ] have mainly focused on the use of GHNs for floating point models. By contrast, we explore the use of GHNs for initializing efficient quantized CNNs and generate a corresponding design space for GHN finetuning. First, we present an in-depth study on the effects of random initialization on quantized accuracy of varying CNN architectures. We show that there are significant differences in final trained, quantized accuracy for different initialization methods. For this random initialization study, we follow the methods of [ 14 ] and perform some layerwise analysis of the trained model distributions. Thus, observing how different initializations can lead to varying ranges and distributions and in turn, varying quantized accuracy. This observation naturally leads us to ask, are there better ways to initialize CNN parameters for training quantization-robust models? To answer this, we present a novel exploration of the use of GHNs to predict quantization-robust efficient CNN parameters (which we will generally refer to as GHN-Q). By finetuning GHNs on a mobile-friendly CNN architecture space, we explore the use of adapting GHNs specifically to target efficient, low-power quantized CNNs. We find that the quantized accuracy of GHN-predicted parameters is surprisingly robust after float32 finetuning on the target design space and even prior to any additional quantization training (of the GHN). Finally, we demonstrate that quantization-aware training of GHNs (which we call GHN-QAT) can significantly improve quantized accuracy of low bitwidth networks such as 4-bit weights and 4-bit activations, and even achieve better-than-random accuracy for 2-bit weights/2-bit activation CNNs. Thus, we contribute a novel, in-depth study of GHNs for quantization-robust initialization of low-precision, efficient CNN architectures. 2 Background and Related Works 2.1 Fixed point quantization for CNNs Various works have explored different quantization algorithms [ 5 , 6 , 15 ] to minimize the loss of information when mapping the weights and activations of a CNN into a discretized space. With these methods, the weights and activations can be represented as an n-bit (most often 8-bit) integer rather than 32-bit floating point numbers. Consequently, simple integer multiply-accumulate (MAC) operations can be performed rather than costly floating point arithmetic. This leads to significant savings in both computation (integer arithmetic) and storage (typically 8-bits per value or less). Mobile hardware accelerators are usually limited in the types of operations that can be massively parallelized for fast execution. Thus, more complex quantization methods are often not supported by existing hardware. As such, other works have focused on quantization-algorithm-specific optimization methods (e.g., targeting 8-bit uniform quantization). These include quantization-aware fine-tuning [ 5 ] and differential optimization of quantization parameters [ 16 , 17 ] , e.g., finding the optimal max/min thresholds of

Read the complete text