ICQuant: Index Coding enables Low-bit LLM Quantization

Xinlin Li †, Osama Hanna ⋄, Christina Fragouli †, Suhas Diggavi † · 2025

Read ICQuant: Index Coding enables Low-bit LLM Quantization by Xinlin Li †, Osama Hanna ⋄, Christina Fragouli †, Suhas Diggavi † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00850.txt.

Introduction

Xinlin Li and co-authors present, in a 2025 machine learning paper, a compression method that squeezes large language models to very few bits per weight. Outlier weights stretch the rounding grid and cause big errors, and taming them usually costs about a bit per weight. Their method borrows index coding from communications to log the outliers cheaply.

The scheme spends only about a third of a bit to halve the rounding range, a saving that matters at two to three bits total. It layers on top of any standard rounding method. On a huge open model at just over two bits, accuracy jumped well past two rival methods and matched a fine-tuned best, with no fine-tuning.

Catalog record

CREATORXinlin Li †, Osama Hanna ⋄, Christina Fragouli †, Suhas Diggavi †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00850

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ICQuant: Index Coding enables Low-bit LLM Quantization Xinlin Li †, Osama Hanna ⋄, Christina Fragouli †, Suhas Diggavi † Abstract The rapid deployment of Large Language Models (LLMs) highlights the need for efficient low-bit post-training quantization (PTQ), due to their high memory costs. A key challenge in weight quantization is the presence of outliers, which inflate quantization ranges and lead to large errors. While a number of outlier suppression techniques have been proposed, they either: fail to effectively shrink the quantization range, or incur (relatively) high bit overhead. In this paper, we present ICQuant, a novel framework that leverages outlier statistics to design an efficient index coding scheme for outlier-aware weight-only quantization. Compared to existing outlier suppression techniques requiring ≈ 1 \approx 1 bit overhead to halve the quantization range, ICQuant requires only ≈ 0.3 \approx 0.3 bits; a significant saving in extreme compression regimes (e.g., 2-3 bits per weight). ICQuant can be used on top of any existing quantizers to eliminate outliers, improving the quantization quality. Using just 2.3 bits per weight and simple scalar quantizers, ICQuant improves the zero-shot accuracy of the 2-bit Llama3-70B model by up to 130% and 150% relative to QTIP ( Tseng et al., 2024b ) and QuIP# ( Tseng et al., 2024a ) ; and it achieves comparable performance to the best-known fine-tuned quantizer ( Malinovskii et al., 2024 ) without fine-tuning. 1 1 1 Code available at: https://github.com/Avery-xl/ICQuant 1 Introduction Despite their success, the accessibility of Large Language Models (LLMs) is curbed by their significant memory and computational power requirements. Weight quantization, a technique that represents model weights using lower precision, is promising to alleviate these demands while preserving the model’s performance. Quantization not only enables a reduced memory footprint, but also accelerates inference time as memory fetch latency is reduced. As a result, quantized models benefit from reduced energy consumption and lower running costs. Quantization opens the road for the deployment of LLMs over resource-constrained devices (for instance, smartphones that have memory and battery constraints), better positions LLMs to serve real-time applications (such as speech recognition and real-time translation that require fast inference times), and overall lowers the deployment cost, leading to more sustainable computing and broader usage. Weight-only post-training quantization (PTQ) has emerged as a popular compression method ( Frantar et al., 2022 ; Lin et al., 2024 ; Zhu et al., 2024 ) . It avoids the heavy computational cost of retraining associated with quantization-aware training (QAT) and enables the on-demand quantization at a required precision. Yet, a major obstacle in achieving low-bit ( \leq 4 bits) weight quantization without significant performance deterioration is the presence of weights with exceptionally high magnitudes that reside in the tails of the weight distribution, known as outliers. Outliers significantly expand the required quantization range, making it hard to cover with a small number of bits without detrimental quantization errors. Given its importance, a number of techniques have been proposed to mitigate the challenge of outliers. For instance, a common technique is to clip the outliers to fixed thresholds. Recent work, such as BitDistiller ( Du et al., 2024 ) and OmniQuant ( Shao et al., 2023 ) , advanced this concept by optimizing the clipping range dynamically. However, the substantial quantization errors associated with outliers limit its practical application. Another prevalent approach is weight grouping ( Frantar et al., 2022 ; Shao et al., 2023 ; Lin et al., 2024 ) . Weight quantization typically occurs at per-(output) channel, and grouping further divides weights into small continuous blocks (e.g., with size 128, 64) to leverage the reduced local ranges. Our statistical analysis (in Section 2 ) reveals that the effectiveness of this method is limited, as outliers frequently persist within individual groups. Furthermore, the storage requirements for group-specific quantization parameters, such as scales, zero-points, and look-up tables, make this approach impractical for non-uniform and vector quantization schemes. More recent approaches include SqueezeLLM ( Kim et al., 2023 ) and SpQR ( Dettmers et al., 2024 ) , which maintain outliers (and/or important weights) in full precision (FP16). While effective, this method also incurs significant storage overhead due to storing the full precision outliers and their indices. QuIP ( Chee et al., 2023 ) proposed an innovative incoherence processing technique, which employs random rotation matrices on both sides of weight matrices to suppress outliers. However, additional matrix operations introduce notable computational overhead during inference. Moreover, when many layers’s weights exhibit independently and identically distributed Gaussian behavior (as observed in Dettmers et al. (2023) ), such rotations often yield small improvement, as discussed in Appendix H.2 . In this paper, we introduce ICQuant, a novel quantization framework that enables us to deal separately with outliers with minimal storage overhead (approximately 0.3 bits per weight). In particular, ICQuant explicitly partitions the weights into outliers and inliers, keeps track of the positions of outliers using an optimized indexing scheme, and uses a different quantization codebook with reduced range for the outlier and inlier weights. A key insight on why our approach works well, is the empirical observation that, within each row of the weight matrices, the positions (indices) of outliers tend to follow a uniform distribution, and thus can be efficiently encoded. We verify this property across multiple models (including the Llama2, 3, 4, and Qwen2.5 families) that have a wide range of scales. We also note that, even if this property were not to hold in other/future models, a one-time random permutation can enforce uniformity without affecting model output and inference speed, and thus ICQuant would still work well. Our contributions include: • We introduce ICQuant, a novel outlier-aware LLM quantization framework that leverages statistical properties of outliers for effective weight quantization. An attractive feature of ICQuant is that it can be universally applied on top of any quantization scheme. • We conduct extensive experiments showing that ICQuant can significantly improve the quantization quality, in 2-4 bits regimes, even of simple scalar quantizers, achieving comparable results to state-of-the-art schemes that rely on computationally intensive vector quantization and expensive fine-tuning procedures. • We analytically derive an upper bound (see Lemma 1 ) on the bits required by our index coding scheme, which closely aligns with our experimental results, and amounts to \approx 0.3 bits per weight. • Our work reveals that the outlier weight spatial distribution is well approximated by a uniform distribution, which may be of independent interest. The rest of the paper is organized as follows. Section 2 presents the statistical properties of outliers that motivate our scheme; Section 3 introduces our design of ICQuant; Section 4 provides our experimental evaluation; Section 5 discusses additional related work, and Section 6 concludes the paper. 2 Statistics of Outliers Although various techniques exist (as described in Section 1 ) to reduce the weight quantization range, the statistical properties of weight outliers, such as their distribution and impact on quantization efficiency, remain poorly understood. This section examines these statistical characteristics in detail. In this paper, we define outliers as the small percentage of weights with high magnitude (i.e., those in the tails of the weight distribution). Specifically, for a weight matrix {\bm{W}}\in\mathbb{R}^{d_{out}\times d_{in}} , outliers are identified as the top \gamma weights (e.g., \gamma=5\% ) with the

Read the complete text