FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs
Xilong Xie, Liang Wang *, Limin Xiao *, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu · 2025
Read FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs by Xilong Xie, Liang Wang *, Limin Xiao *, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19746.txt.
Introduction
Xilong Xie and co-authors (2025) present FineQ, a way to shrink large language models with less damage to their accuracy. Quantization stores weights with fewer bits, but going very low usually hurts. Mixed precision helps, yet grouping weights coarsely wastes memory or accuracy. FineQ splits weights into finer clusters and studies where the outlier values sit inside each cluster. Outliers get three bits of protection. A packing scheme keeps memory reads aligned and fast.
The authors also designed a chip accelerator for the scheme. Temporal coding simplifies the multipliers in its computing grid. At nearly the same average bit width, FineQ kept more accuracy than the best rival method. The accelerator used up to 1.79 times less energy and shrank the array area by about sixty percent. The authors present the pair as one design: the algorithm and the hardware were shaped together, which is where the gains came from.
Catalog record
| CREATOR | Xilong Xie, Liang Wang *, Limin Xiao *, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19746
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs Thanks: * Liang Wang and Limin Xiao are corresponding authors of this paper. This work was supported in part by the National Science and Technology Major Project under Grant No. 2022ZD0117602; in part by the National Natural Science Foundation of China under Grants 62272026 and 62104014; in part by the Fundamental Research Funds for the Central Universities; in part by State Key Laboratory of Complex & Critical Software Environment under Grant No. CCSE-2024ZX-10. Xilong Xie, Liang Wang *, Limin Xiao *, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu Abstract Large language models (LLMs) have significantly advanced the natural language processing paradigm but impose substantial demands on memory and computational resources. Quantization is one of the most effective ways to reduce memory consumption of LLMs. However, advanced single-precision quantization methods experience significant accuracy degradation when quantizing to ultra-low bits. Existing mixed-precision quantization methods are quantized by groups with coarse granularity. Employing high precision for group data leads to substantial memory overhead, whereas low precision severely impacts model accuracy. To address this issue, we propose FineQ, software-hardware co-design for low-bit fine-grained mixed-precision quantization of LLMs. First, FineQ partitions the weights into finer-grained clusters and considers the distribution of outliers within these clusters, thus achieving a balance between model accuracy and memory overhead. Then, we propose an outlier protection mechanism within clusters that uses 3 bits to represent outliers and introduce an encoding scheme for index and data concatenation to enable aligned memory access. Finally, we introduce an accelerator utilizing temporal coding that effectively supports the quantization algorithm while simplifying the multipliers in the systolic array. FineQ achieves higher model accuracy compared to the SOTA mixed-precision quantization algorithm at a close average bit-width. Meanwhile, the accelerator achieves up to 1.79× energy efficiency and reduces the area of the systolic array by 61.2%. I Introduction Large language models (LLMs) based on transformers [ 1 ] have demonstrated remarkable performance in various natural language processing tasks through their enormous model size. However, LLMs, as one of the most significant computational workloads today, are challenging to deploy on edge devices due to their extensive memory and computational resources required [ 29 ] . For instance, deploying the LLaMA-2-70B in FP16 requires at least 140 GB of memory [ 2 ] , markedly exceeding the 80 GB capacity of an A100 GPU. Weight quantization [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 4 ] is one of the most effective ways to reduce the memory consumption of LLMs while preserving performance, achieved by storing parameters in a low-precision representation. Fig. 1: Perplexity of LLaMA-2-7B on C4 under differnet bit-widths. Lower perplexity means better model accuracy. Round-to-nearest (RTN), GPTQ, PB-LLM (10% weight of FP16), and OWQ(g=128) suffer from accuracy loss at ultra-low bits. FineQ demonstrates better performance than these methods. To enable the future deployment of LLMs on edge devices [ 28 ] , more aggressive quantization and compression techniques will be essential. Existing quantization methods are still inadequate for this purpose, as reducing precision to ultra-low bits( < 3 bits) leads to a significant degradation in model accuracy. Currently, weight quantization techniques primarily fall into single-precision quantization and mixed-precision quantization. Several single-precision quantization methods, such as RTN [ 4 ] , GPTQ [ 5 ] , have successfully quantized the FP16 LLMs into 4-bit or 3-bit formats while maintaining acceptable performance. Regrettably, as shown in Figure 1 , advanced single-precision quantization methods for LLMs experience notable performance degradation under ultra-low bit quantization. This observation underscores the challenges that existing methods encounter when quantizing models to ultra-low bits. The substantial loss of model accuracy in single-precision quantization is mainly due to the inadequate handling of weight outliers [ 6 ] . Outliers are identified by a small set of values with exceptionally high magnitudes [ 27 ] . Single-precision quantization methods indiscriminately clip both outliers and normal values, leading to notable drops in model accuracy [ 7 ] . To maintain model accuracy, recent works apply mixed-precision quantization methods. Fig. 2: (a) The transformer block architecture. (b) Memory layout for serving a 13B-parameter LLM on the NVIDIA A100 (40GB) [ 17 ] . For instance, OWQ [ 9 ] and LLM-MQ [ 10 ] quantize the majority of normal weight values to 2 bits while representing outliers in FP16 format. PB-LLM [ 11 ] binarizes a subset of weights in LLMs while preserving outliers with high precision. Though existing mixed-precision quantization algorithms preserve model accuracy to some extent, they still demonstrate shortcomings in the handling of outliers. In existing mixed-precision quantization methods [ 25 , 9 , 11 , 10 ] , weights are quantized in coarse-grained groups, with certain groups containing weights with large range. Therefore, these methods fail to adequately protect outliers and struggle to achieve a trade-off between memory overhead and model accuracy. Moreover, these methods treat outliers as sparse matrices, necessitating additional sparse formats and complex control logic to handle irregular computations and memory access, leading to lower computational efficiency. In this work, we introduce FineQ, a low-bit fine-grained mixed-precision quantization method of LLMs with software-hardware co-design. We aim to achieve a balance between model accuracy and memory overhead by adopting a fine-grained outlier protection strategy. Specifically, we propose a series of innovative mechanisms and architectures, including a fine-grained mixed-precision quantization algorithm, an intra-cluster outlier protection mechanism, and a temporal coding-based hardware accelerator. The main contributions of this paper are summarized as follows. 1) We propose a fine-grained mixed-precision quantization algorithm that does not require retraining. The algorithm partitions model weights into smaller clusters and considers the distribution of outliers within each small cluster, enabling quantization to lower bit-width while maintaining model accuracy. 2) We introduce an intra-cluster outlier protection mechanism. By observing the distribution of weight outliers and the effects of various quantization bit-widths on model accuracy, we quantize the majority of normal values to 2 bits, apply 3-bit protection for outliers, and use an innovative memory alignment scheme that integrates encoding information with the data. 3) We present an efficient hardware accelerator based on temporal coding to support the quantization algorithm. Furthermore, temporal coding allows for efficient bitstream generation from weights [ 3 ] , significantly simplifying the multiplier design in the systolic array. The accelerator achieves up to 1.79× energy efficiency and reduces the systolic array area by 61.2%. II Background and Motivation II-A Transformer-Based Large Language Models Fig. 3: Three observations about hardware-software co-design of mixed-precision quantization. The transformer model serves as the backbone architecture for existing LLMs [ 1 ] , enabling efficient processing for tasks such as natural language understanding and generation [ 16 , 15 ] . As illustrated in Figure 2 (a), each transformer block contains self-attention and feed-forward network (FFN). The self-attention mechanism quantifies the degree of association among different positions within a single sequence and it consists of QKV generation, attention, and a linear layer. The FFN consists of two linear layers and an activation function. In recent years, the scale of large language models has continued to grow [ 23 ] . Figure 2 (b) illustrates the memory distribution for a 13B-parameter LLM on an