NeUQI: Near-Optimal Uniform Quantization Parameter Initialization

Li Lin, Xinyu Hu, Xiaojun Wan · 2025

Read NeUQI: Near-Optimal Uniform Quantization Parameter Initialization by Li Lin, Xinyu Hu, Xiaojun Wan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17595.txt.

Introduction

Shrinking a large language model so it fits a laptop or a home graphics card means rounding its numbers to fewer bits. Li Lin, Xinyu Hu, and Xiaojun Wan study a step in that shrink that past work has largely skipped: where to set the rounding scale at the start. Their 2025 method is called NeUQI.

The old rule takes the plain span from the least value to the most, which can start the shrink at a poor point. The new method seeks a near-best start with modest effort. Most past gains came from the shrink method itself, not from the start values. A better start costs little and can lift the whole result. The work is aimed at low-bit models that must run well on modest gear.

Catalog record

CREATORLi Lin, Xinyu Hu, Xiaojun Wan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NeUQI: Near-Optimal Uniform Quantization Parameter Initialization Li Lin, Xinyu Hu, Xiaojun Wan Abstract Large language models (LLMs) achieve impressive performance across domains but face significant challenges when deployed on consumer-grade GPUs or personal devices such as laptops, due to high memory consumption and inference costs. Post-training quantization (PTQ) of LLMs offers a promising solution that reduces their memory footprint and decoding latency. In practice, PTQ with uniform quantization representation is favored for its efficiency and ease of deployment since uniform quantization is widely supported by mainstream hardware and software libraries. Recent studies on ≥ 2 \geq 2 -bit uniform quantization have led to noticeable improvements in post-quantization model performance; however, they primarily focus on quantization methodologies, while the initialization of quantization parameters is underexplored and still relies on the suboptimal Min-Max strategies. In this work, we propose NeUQI , a method devoted to efficiently determining near-optimal initial parameters for uniform quantization. NeUQI is orthogonal to prior quantization methodologies and can seamlessly integrate with them. The experiments with the LLaMA and Qwen families on various tasks demonstrate that our NeUQI consistently outperforms existing methods. Furthermore, when combined with a lightweight distillation strategy, NeUQI can achieve superior performance to PV-tuning, a much more resource-intensive approach. 1 Introduction In recent years, large language models (LLMs) like ChatGPT [ 1 ] have rapidly emerged, demonstrating strong capabilities across various tasks, including open-ended writing, knowledge-based question answering, and code generation. Given the high API costs of proprietary LLMs and the growing performance of open-source alternatives like LLaMA [ 28 , 10 ] and Qwen [ 32 ] families, there is a rising preference for deploying open-source LLMs locally. However, the deployment of large-scale models (e.g., LLaMA 3 70B) is often limited by compute resources and inference efficiency, especially on personal devices or consumer-grade GPUs like a single RTX 4090 with 24GB of memory. In this context, post-training quantization (PTQ) [ 15 ] , particularly using a uniform quantization scheme, offers a practical solution by converting model weights from bfloat16 to low-bit-width integer formats such as int4/3/2 , significantly reducing memory usage and inference latency. Prior work on uniform PTQ mainly focuses on two directions. The first involves weight and activation processing to reduce quantization difficulty, such as smoothing via rotation transformations [ 5 , 29 , 3 ] , integrating rotations into model architectures [ 21 , 13 ] , and reducing weight magnitude [ 34 ] . The second emphasizes block-wise and global fine-tuning, including multi-block training [ 7 ] , rotation-based fine-tuning [ 21 , 13 ] , and direct optimization over quantized weights [ 22 ] . Although these studies have made notable progress, they still adopt the conventional Min-Max initialization, which merely determines quantization parameters solely based on the extreme values in the weights while ignoring weight distribution and importance. As revealed in LeanQuant [ 35 ] , such initialization leads to a mismatch with the quantization objective in the commonly adopted GPTQ framework [ 8 ] . Even under ideal conditions, specifically a uniform distribution, Min-Max remains suboptimal, as analyzed in Appendix A . However, little research has focused on improving quantization initialization. Only recently has LeanQuant made a preliminary attempt by introducing a loss-aware strategy. But it still inherits a key limitation of Min-Max: restricting the zero-point to integers, which reduces the flexibility of uniform quantization and results in suboptimal initialization. To address this gap, we propose NeUQI, a novel initialization method for uniform quantization, which removes the integer constraint on the zero-point, thus enabling a more flexible parameter space. We first apply a diagonal approximation to the quantization reconstruction objective and then decompose it into two tractable subproblems: (1) determining the optimal zero-point under a fixed scale and (2) finding the optimal scale given its corresponding zero-point. Both are efficiently solvable by exploiting the objective’s structure. More importantly, NeUQI is plug-and-play and orthogonal to quantization methods used in the calibration and fine-tuning stages. When combined with our NeUQI, the performance of quantized models significantly improves and even surpasses non-quantized models of comparable memory usage, as illustrated in Figure 1 . Furthermore, only applying a lightweight knowledge distillation, NeUQI can outperform PV-tuning [ 22 ] —a more sophisticated and resource-intensive fine-tuning method. It underscores the critical role of proper initialization: poor initial conditions are difficult to fix even with heavy fine-tuning, while a well-initialized model can achieve better performance with significantly less effort. Figure 1: Perplexity on C4 (left) and average accuracy across five common benchmarks (right) with Qwen 2.5 family, plotted against the base-2 logarithm ( \log_{2} ) of model memory usage on the x-axis. The results include the non-quantized models (BF16) and the quantized models using commonly-used GPTQ, or our proposed NeUQI (both 3-bit). More details and results are shown in Section 5 . The main contributions of our work are summarized as follows: 1. We revisit the conventionally adopted Min-Max quantization initialization and identify its fundamental limitation: the restriction of the zero-point to integers, which leads to reduced flexibility and suboptimal initialization. 2. We propose NeUQI, a novel initialization method for uniform quantization that relaxes the zero-point to a continuous variable and exploits the structure of the optimization landscape. NeUQI can be directly combined with existing quantization methods to enhance their performance. 3. Experiments with LLaMA and Qwen families of different sizes on various tasks demonstrate that NeUQI consistently outperforms existing methods. When further combined with a lightweight distillation strategy, NeUQI even surpasses relatively more resource-intensive PV-tuning. 2 Related Work Uniform Quantization In the 2-bit to 4-bit quantization regime, prior studies have explored many post-training quantization techniques to mitigate accuracy degradation. Early methods like AWQ [ 19 ] and SmoothQuant [ 31 ] applied mathematically equivalent transformations using diagonal matrices on activations and their inverses on weights, balancing quantization difficulty and laying the groundwork for later approaches. Subsequent methods generalized the diagonal matrix to invertible forms for better smoothing, though activation-side transformations require online computation, leading to efficiency-focused designs. For example, QuIP [ 5 ] uses Kronecker products of two small rotations; DuQuant [ 18 ] adopts block-diagonal matrices with repeated rotation; QuIP# [ 29 ] and QuaRot [ 3 ] use Hadamard transforms; FrameQuant [ 2 ] applies a Fourier-based transformation variant. SpinQuant [ 21 ] and OSTQuant [ 13 ] introduce learnable rotations integrated into the model to avoid runtime overhead. Alternatively, MagR [ 34 ] optimizes weights while preserving model functionality to reduce quantization difficulty. In addition, there are fine-tuning-based approaches such as CBQ [ 7 ] , which performs block-wise tuning, and PV-tuning [ 22 ] , which jointly optimizes continuous and discrete parameters. Quantization Parameter Initialization Most uniform quantization methods rely on the conventional Min-Max initialization, which determines quantization parameters by the extreme weight values. Although Min-Max is straightforward and efficient, this heuristic neglects the impact of initialization on the final model performance. Some methods, such as MagR and OmniQuant [ 27 ] , introduce shrink-scale variants of Min-Max to suppress outliers and improve precision, but they remain simple extensions of the original method. LeanQuant [ 35 ] makes a significant step forward by proposing an optimization-based initialization with grid search. However, its optimization relies on enumerating different Min-Max value pairs,

Read the complete text