InfoQuant : Shaping Activation Distributions for Low-Bit LLM Quantization

Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang · 2026

Read InfoQuant : Shaping Activation Distributions for Low-Bit LLM Quantization by Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26175.txt.

Introduction

Li and eight colleagues recast activation changes as quantiser-facing distribution design for low-bit quantisation. Low-bit quantisation slows fast deployment, and the trouble is not only outliers. They often mismatch low-bit uniform quantisers. Current post-training methods suppress peaks, balance channels or cut error, yet rarely specify an easy-to-quantise target. They can look smoother yet suffer large error, with wide ranges or values collapsed near the mean. The authors analyse error from an information view, designing transforms that shape activations into friendlier shapes.

Catalog record

CREATORKe Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55812
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InfoQuant : Shaping Activation Distributions for Low-Bit LLM Quantization Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang Abstract Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant , a train-free method that employs Peak Suppression Orthogonal Transformation ( PSOT ) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. 1 1 1 Code is available at: github.com/LLIKKE/InfoQuant . 1 Introduction Post-training quantization (PTQ) is one of the most practical ways to reduce the memory and compute cost of large language model (LLM) inference. Its main challenge, however, lies in low-bit activation quantization. Unlike weights, LLM activations often contain a small number of dominant coordinates that enlarge the quantization range and force round-to-nearest quantization to map many normal values to the same few levels. This mismatch becomes particularly severe in 4-bit settings, where limited quantization levels leave little room to preserve both rare extremes and dense central values. Recent PTQ methods increasingly address this problem through activation transformations before quantization. SmoothQuant 33 migrates activation difficulty into weights through diagonal scaling, while QuaRot 3 and SpinQuant 20 use orthogonal rotations to redistribute activation energy; other methods further introduce more flexible affine transformations or reconstruction objectives 21 ; 29 . Although these methods differ in form, they share the same practical role: they change the activation distribution seen by the quantizer. Yet most of them are motivated by suppressing outliers, balancing channels, or reducing reconstruction error, rather than by defining what transformed distribution a low-bit quantizer can represent well. Consequently, as illustrated in Figure 1 , they may reduce visible peaks without fully improving discretizability, or preserve small numerical error while still collapsing distributional resolution. The central question, therefore, is not only how to transform activations, but what transformed activation distribution is actually quantization-friendly. Figure 1: Activation distributions of the LLaMA-2 7B, layer4 q/k/v_proj input under three transformations: the original activations (left), Hadamard rotation (center), and the learned rotation from PSOT (right). Compared with the original and Hadamard-rotated activations, PSOT produces a more quantization-friendly distribution with a narrower numerical range and larger normalized dispersion. Here, b_{n} denotes the standard deviation after infinity-norm normalization, and a smaller \lambda=\bar{s}/b_{n} indicates a lower normalized quantization-error bound, where \bar{s} is the range-normalized quantization step size. We address this gap by recasting activation transformation as quantizer-facing distribution design. From an information-theoretic perspective, we analyze how quantization error depends on the activation distribution after transformation. Our theoretical and empirical study shows that lower quantization error is associated with two complementary properties: a smaller numerical range and greater dispersion within that range. This result reframes the role of activation transformation. Rather than merely suppressing outliers or minimizing heuristic reconstruction losses, a PTQ method should explicitly shape activations into distributions that are easier for a low-bit quantizer to preserve. Since LLM activations are typically bell-shaped and often contain outliers 18 , they are naturally misaligned with this target, which explains why low-bit activation quantization remains difficult. Guided by this principle, we introduce InfoQuant , a train-free PTQ method that learns orthogonal transformations to produce more quantization-friendly activation distributions. Its core component is Peak Suppression Orthogonal Transformation ( PSOT ), which applies an activation-wise peak suppression objective to reduce the numerical range while increasing normalized dispersion. We further introduce adaptive outlier-token selection to improve optimization robustness, and learn activation clipping parameters to refine the final quantization range after the distribution has been reshaped. Although InfoQuant is designed around activation optimization, it remains compatible with standard weight quantization pipelines. Overall, our contributions can be summarized as follows: • We introduce an information-theoretic framework for understanding activation quantization error and show, both theoretically and empirically, that quantization-friendly distributions should have a smaller numerical range and greater dispersion. • We introduce InfoQuant , a hardware-efficient and train-free PTQ method centered on learned orthogonal activation shaping, together with adaptive outlier-token selection and learnable activation clipping for robust calibration. • We demonstrate that activation-distribution optimization yields strong empirical and practical gains. In the W4A4KV4 setting, LLaMA-2 (7B, 13B, 70B) and LLaMA-3 (8B, 70B) retain an average of 97\% of their original performance, and the 70B model can be quantized using only 24 GB of GPU memory. 2 Related Work Post Training Quantization for LLMs. PTQ is an efficient and widely used approach for compressing LLMs. Due to the flatness and uniform distribution of LLM weights, weight-only quantization typically results in minimal performance degradation. GPTQ 9 uses Hessian-based error compensation to enable high compression with low accuracy loss. AWQ 17 and OWO 14 further improve performance by mitigating the effects of activation outliers. QuIP ( 6 ) and QuIP# ( 31 ) apply random Hadamard transforms for incoherent processing and employ vector quantization on weights, achieving better performance. In contrast, activation quantization remains more challenging due to the presence of rare but extreme outliers ( 32 ; 33 ) , which can disproportionately affect accuracy. Transformation-based Methods. These methods more effectively redistribute activation outliers across channels. Channel scaling ( 33 ) shifts part of this burden to weights, OmniQuant ( 28 ) and LRQuant ( 36 ) optimize scaling parameters via MSE minimization. However, recent work ( 34 ) shows channel scaling alone fails under 4-bit settings, leading to notable degradation. AffineQuant ( 21 ) learns affine transformations to precondition activations. However, due to the significant overhead of full-size matrix multiplication, AffineQuant can only apply affine transformations to a small fraction of linear layers. FlatQuant ( 29 ) reduces this cost via Kronecker decomposition, applying affine transformation to every linear layer. Leveraging computational invariance ( 2 ) , orthogonal transforms can be applied to weights and between-block activations without extra inference overhead. QuaRot ( 3 ) uses randomized Hadamard transforms to remove outliers. SpinQuant ( 20 ) further optimizes learnable orthogonal matrices on the Stiefel manifold with task loss (e.g., cross-entropy) to find stable transformations. OSTQuant 13 combines channel scaling with orthogonal transforms and uses end-to-end distillation from original outputs to boost quantization. Kurtail ( 1 ) facilitates quantization by controlling the kurtosis to make the distribution more uniform. BASE-Q ( 12 ) introduces an additional bias term to balance the mean

Read the complete text