TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

Zukang Xu, Xing Hu, Dawei Yang · 2026

Read TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization by Zukang Xu, Xing Hu, Dawei Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19561.txt.

Introduction

Xu, Hu and Yang propose TORQ, a training-free quantization method that rescues MXFP4 activation accuracy. Direct MXFP4 activation quantization degrades accuracy badly through two imbalances: uneven variance across blocks and uneven codebook use within blocks. TORQ rotates activations at two levels: energy spreads across blocks by the Schur-Horn theorem, and codebook use spreads within blocks. On Qwen3-32B it brings WikiText perplexity to 8.43, near the 7.61 full-precision figure, and lifts average accuracy to 73.63 percent from 38.40 percent, near the 74.82 percent mark.

Catalog record

CREATORZukang Xu, Xing Hu, Dawei Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56793
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19561

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization Zukang Xu, Xing Hu, Dawei Yang Abstract As Large Language Models (LLMs) advance toward practical deployment, the Microscaling FP4 (MXFP4) format has emerged as a cornerstone for next-generation low-bit inference, owing to its ability to balance high dynamic range with hardware efficiency. However, directly applying MXFP4 to LLM activation quantization inevitably leads to significant accuracy degradation. In this paper, we theoretically analyze the error structure of MXFP4 activation quantization, revealing that the root cause of this performance drop lies in two structural imbalances between activation distributions and the MXFP4 block floating-point format: (1).Extreme Inter-block Variance Imbalance: Under fixed-block microscaling, inter-block variance is highly uneven. Specifically, a minority of high-variance blocks dominate the total quantization error and force the shared scaling factor upward, resulting in the excessive coarsening of numerous small-magnitude activations within the block. (2).Intra-block Codebook Utilization Imbalance: The heavy-tailed distribution of real-world activations severely mismatches the MXFP4 geometric progression codebook, leading to extremely low effective codeword utilization and leaving significant representation capacity idle. To address these challenges, we propose TORQ (Two-level Orthogonal Rotation for MXFP4 Quantization), a training-free Post-Training Quantization (PTQ) framework designed to reshape the geometric properties of the activation space through optimal coordinate transformations. At the macroscopic level, TORQ leverages the Schur-Horn theorem to redistribute activation energy via inter-block orthogonal rotation. Mathematically formalized as variance equalization, this process acts as a soft outlier suppression mechanism—preventing high-variance blocks from driving up the shared scaling factors and thereby preserving the precision of small-magnitude elements. At the microscopic level, TORQ employs maximum-entropy-guided intra-block rotation to alleviate ”codebook collapse.” Actively dispersing the highly concentrated activation distribution, it effectively ”activates” underutilized codewords to maximize the MXFP4 codebook’s information capacity. Experiments on various mainstream LLMs such as LLaMA3 and Qwen3 show that TORQ significantly improves the accuracy of MXFP4 activation quantization compared to existing SOTA methods: on the Qwen3-32B model, the perplexity on WikiText is reduced to 8.43 (7.61 for BF16), and the average accuracy is increased from 38.40% with direct RTN to 73.63% (74.82% for BF16), successfully bridging the gap between 4-bit floating-point quantization and full-precision inference. † † affiliationnotice: Equal contribution † Corresponding author. 1 Introduction With the widespread adoption of Large Language Models (LLMs) across various domains ( Yang et al. 2025 ; Dubey et al. 2024 ) , their massive parameter counts and the memory bandwidth demanded during inference have become major bottlenecks for practical deployment ( Czakó et al. 2025 ) . To run LLMs efficiently on resource-constrained edge devices or in large-scale data centers, Post-Training Quantization (PTQ) has emerged as the industry standard paradigm ( Ashkboos et al. 2024a ; Xu et al. 2025b ; Xu et al. 2025a ; Yu et al. 2025 ; Hu et al. 2025a ) . While traditional INT4 quantization has achieved significant success in weight compression, its limited dynamic range struggles to accommodate the long-tail outliers prevalent in Transformer activations ( Czakó et al. 2025 ) . Consequently, the industry is shifting toward low-precision floating-point formats with higher dynamic ranges through hardware co-design. Notably, the MXFP4 (e2m1) standard introduced by the Open Compute Project (OCP), which employs Block-wise Shared Exponent technology, offers representation capabilities approaching FP8 while maintaining 4-bit compression rates. Supported natively by next-generation hardware such as NVIDIA Blackwell and AMD Ryzen AI ( Hu et al. 2025b ; Xiao et al. 2025 ) , MXFP4 is widely regarded as the cornerstone of next-generation low-bit inference. Figure 1 : Performance of Different Quantization Methods on LLaMA3-8B and LLaMA3-70B. Despite the theoretical advantages of MXFP4, our empirical analysis reveals that directly applying it to LLM activation quantization encounters severe accuracy bottlenecks. State-of-the-art quantization methods, such as SingleQuant ( Xiao et al. 2025 ) , DARTQuant ( Shao et al. 2025a ) , FlatQuant ( Sun et al. 2024 ) , and OSTQuant ( Hu et al. 2025b ) , have successfully addressed W4A4 integer quantization through rotation techniques. However, these methods are primarily designed to suppress outliers to fit the uniform grid of integers. This design overlooks the unique Block Floating Point (BFP) topological structure of MXFP4. As a result, simply transferring rotation strategies optimized for integers to MXFP4 often fails to exploit the potential of the floating-point format and may even be counterproductive. Figure 2 : MXFP4 Quantization of each micro block’s variance distribution: (a) Variance distribution of original data without TORQ, (b) Variance distribution after TORQ processing To address this challenge, this paper conducts an in-depth analysis of the error structure of MXFP4 activation quantization, revealing that the root cause of accuracy degradation lies in two structural conflicts between the activation distribution and the MXFP4 format: (1).Extreme Inter-block Variance Imbalance: The quantization accuracy of MXFP4 relies heavily on the block-wise shared scaling factor. When the activation vector is partitioned into fixed blocks (e.g., Block Size=32), as shown in Figure. 2 , block variance exhibits a significant heavy-tailed distribution. A minority of high-energy blocks not only dominate the total Mean Squared Error (MSE) but also force the shared exponent to rise sharply, causing over 90% of small-magnitude elements within the same block to be zeroed out due to insufficient resolution. (2).Intra-block Codebook Utilization Imbalance (Codebook Collapse): The e2m1 codebook design of MXFP4 assumes data follows an ideal log-normal distribution. However, real-world activation distributions are often highly concentrated near zero, causing the vast majority of values to fall within the minimum quantization interval, while numerous codewords with strong representation capabilities for large values remain idle (i.e., Codebook Collapse). As shown in Figure. 3 , our statistics show that approximately 50% of the codeword range is activated by only 10% of the data in actual inference, resulting in severe bit waste. Figure 3 : Histogram statistics of each codeword quantified by FP4(e2m1):(a) Codeword occupancy of the original data without TORQ, (b) Codeword occupancy after TORQ processing Based on these key insights, we propose TORQ (Two-level Orthogonal Rotation for MXFP4 Quantization). This is a retraining-free PTQ framework whose core philosophy is: Instead of passively adjusting quantization parameters to fit ill-conditioned activation distributions, we should actively rotate the activation coordinate system to align its distribution characteristics with the structural properties of MXFP4. TORQ decouples the optimization problem into two mathematically orthogonal sub-problems: (1).Macro-Equilibrium Rotation: To address inter-block variance imbalance, TORQ constructs an inter-block orthogonal transformation based on the Schur-Horn theorem to uniformly distribute activation energy across all blocks. We theoretically prove that this variance-equalized configuration is the optimal solution for minimizing global quantization error. (2).Micro-Alignment Rotation: To address low intra-block codebook utilization, TORQ introduces the principle of maximum entropy to execute orthogonal rotation within groups. This reshapes the local data distribution to ”flow” uniformly into each codeword interval in the FP4 space, thereby maximizing effective information capacity. TORQ enables the closed-form construction of rotation matrices using only a minimal volume of calibration data. Furthermore, the rotation matrices incorporated during the inference phase feature an extremely compact parameter footprint, imposing negligible additional computational overhead.Experimental evaluations on mainstream large language models (LLMs), including LLaMA and Qwen, demonstrate that TORQ achieves a significant performance superiority over prevailing state-of-the-art (SOTA) approaches. As illustrated in Figure. 1 , this method consistently outperforms existing advanced quantization schemes across the entire Llama3 model series.

Read the complete text