PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models
Yongsen Cheng, Kai Liu, Kaiwen Tao, Junxian Li, Zhixin Wang, Zhikai Chen, Renjing Pei, Yulun Zhang · 2026
Read PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models by Yongsen Cheng, Kai Liu, Kaiwen Tao, Junxian Li, Zhixin Wang, Zhikai Chen, Renjing Pei, Yulun Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09503.txt.
Introduction
Cheng and seven colleagues attack the quality loss when large diffusion models are squeezed into very low bit precision. Post training quantization shrinks a trained model with no retraining. But grouping channels with very unlike values under one shared scale lets outliers rule and errors balloon. Their PermuQuant frame names channel order as a missed lever. It sorts channels by a joint second moment rule, putting channels with like activation and weight values together.
A tuning data rule applies a reorder only when it truly cuts error on tuning data. The chosen reorders fold into nearby modules or weights offline, so no runtime reorder cost stays. Across large diffusion models PermuQuant beats past baselines. On one flagship model it speeds single steps a lot and cuts memory more than threefold under four bit quantization, keeping image quality far better than rivals.
Catalog record
| CREATOR | Yongsen Cheng, Kai Liu, Kaiwen Tao, Junxian Li, Zhixin Wang, Zhikai Chen, Renjing Pei, Yulun Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09503
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models Yongsen Cheng, Kai Liu, Kaiwen Tao, Junxian Li, Zhixin Wang, Zhikai Chen, Renjing Pei, Yulun Zhang Abstract Large-scale visual generative models have achieved remarkable performance. However, their high computational and memory costs make deployment challenging in resource-constrained scenarios, such as interactive applications and personal single-GPU usage. Post-training quantization (PTQ) offers a practical solution by compressing pretrained models without expensive retraining. However, existing PTQ methods still suffer from severe quality degradation under extremely low-bit settings. In this paper, we identify channel ordering as an important but underexplored factor in per-group quantization. In this setting, each contiguous group shares one quantization scale. When channels with very different statistics are placed in the same group, the scale can be dominated by outliers and cause large quantization errors. Based on this observation, we propose PermuQuant , a simple and effective PTQ framework for low-bit diffusion models. PermuQuant sorts channels by a joint second-moment criterion before per-group quantization, placing channels with similar activation and weight statistics into the same group. It further uses a calibration-based acceptance rule to apply reordering only when the selected permutation reduces quantization error on calibration data. The selected permutations are absorbed into adjacent modules or applied to weights offline, avoiding explicit runtime permutation operations. Extensive experiments on multiple large diffusion models show that PermuQuant consistently reduces quantization error and outperforms existing PTQ baselines. On FLUX.1-dev with an RTX 5090, PermuQuant achieves up to a 1.7 × \times single step speedup and reduces the DiT memory footprint by 3.5 × \times under W4A4 NVFP4 quantization. Code will be available at https://github.com/yscheng04/PermuQuant . FLUX.1-dev BF16 (50 Steps) DiT Mem.: 22.2 GiB E2E Lat.: 81.4 s NF4 (W4A16) LPIPS: 0.263 DiT Mem.: 5.6 GiB (4.0 \times Less) E2E Lat.: 23.3 s (3.8 \times Faster) NVFP4 (W4A4) LPIPS: 0.596 DiT Mem.: 6.3 GiB (3.5 \times Less) E2E Lat.: 13.1 s (6.2 \times Faster) PermuQuant NVFP4 (W4A4) LPIPS: 0.186 DiT Mem.: 6.3 GiB (3.5 \times Less) E2E Lat.: 13.0 s (6.3 \times Faster) Prompt: A beautiful anime-style girl, upper body, with long light-blue hair and blue eyes, wearing a white and blue outfit, holding a huge neon sign that says “Permutation Matters!” FLUX.1-dev BF16 (50 Steps) OmniQuant (W3A3) LPIPS: 0.673 ConvRot (W3A3) LPIPS: 0.428 PermuQuant INT (W3A3) LPIPS: 0.327 Prompt: A kingfisher swooping to break digitised binary water, captures information in beak Figure 1: PermuQuant is a post-training quantization framework for low-bit diffusion models. In the W4A4 setting, it achieves 3.5 \times DiT memory reduction and 6.3 \times speedup on a single RTX 5090 32GB GPU by eliminating CPU offloading. In the challenging W3A3 setting, PermuQuant still produces visually clean results with faithful details, significantly outperforming other baselines. 1 Introduction Diffusion models have become a dominant paradigm for visual generation. By learning to reverse a gradual denoising process, they can generate high-fidelity images with strong diversity [ 49 , 12 ] . Recent advances further improve their sampling process, noise schedules, and continuous-time formulations [ 50 , 37 , 52 ] . Together with classifier-free guidance and latent diffusion, these methods have enabled powerful text-to-image generation systems [ 13 , 43 , 40 ] . Meanwhile, the backbone of diffusion models has shifted from convolutional UNets to scalable transformer architectures [ 12 , 55 , 4 , 15 ] . This trend greatly improves generation quality. However, it also increases the memory footprint and computational cost of inference. Since diffusion sampling requires repeated network evaluations, the cost becomes especially high for large models and high-resolution generation. Model quantization [ 36 ] has emerged as a widely adopted technique to compress floating-point parameters and activations into lower-precision formats, thereby reducing memory overhead and accelerating inference. The resulting speedup stems from both diminished cache loading costs and faster low-bit arithmetic. Existing quantization approaches are broadly categorized into two paradigms: post-training quantization (PTQ) and quantization-aware training (QAT). While QAT explicitly models quantization effects during training and achieves strong performance at extremely low bit-widths [ 41 , 5 ] , it typically requires full or partial model retraining, incurring substantial computational and data costs. In contrast, PTQ operates directly on pretrained checkpoints without additional training, making it considerably more resource-efficient. Recent advances in PTQ [ 54 , 1 , 18 , 23 ] have demonstrated that competitive accuracy can be preserved even at low bit-widths. These methods offer a highly practical pathway for deploying large-scale visual models in resource-constrained environments. For example, interactive applications require low-latency inference. Figure 2 : Example of per-group quantization. Quantization error is greatly affected by channel orders. Despite this progress, extremely low-bit weight-activation quantization remains challenging (see Fig. PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models ). In particular, 3-bit quantization can cause severe degradation in image quality and text-image alignment. A common remedy is per-group quantization, which partitions each token or channel into small contiguous groups of equal size and assigns each group its own scale. This strategy gives finer-grained scaling than per-token or per-channel quantization. The key insight of this work is simple: per-group quantization makes channel order important . As illustrated in Fig. 2 , since groups are formed over contiguous channels, the order decides which channels share a scale. This matters because each group scale is controlled by the largest-magnitude values in the group. If statistically mismatched channels are grouped together, outlier channels dominate the shared scale. The remaining channels are then quantized on an overly coarse grid, which increases quantization error. By contrast, placing channels with similar statistics into the same group produces more balanced and better matched scales, thereby reducing quantization error. In this paper, we propose PermuQuant , a simple and effective PTQ framework for low-bit diffusion models. PermuQuant exploits the group structure of per-group quantization by reordering channels before quantization. It uses channel second moments estimated from calibration data to construct more favorable contiguous groups. To formalize this strategy, we show that second-moment based sorting minimizes an upper bound of the expected quantization error. We further derive an approximation guarantee for the expected error objective under the uniform noise approximation [ 33 ] . These results provide a theoretical justification for using channel second moments to guide reordering. To make reordering practical for linear layers, firstly , PermuQuant jointly considers activation and weight statistics. This is necessary because preserving the function of a linear layer requires applying the same permutation to both sides. We therefore use a joint second-moment criterion to select the channel order. Secondly , not all layers benefit equally from reordering. To address this issue, we introduce a calibration-based acceptance rule. It applies reordering only when the selected permutation sufficiently reduces the quantization error over calibration data. Thirdly , another practical challenge is that explicit channel permutation may introduce extra runtime operations. To avoid this cost, we apply the permutation to weight matrix offline and absorb the activation-side permutation into adjacent modules. This design enables PermuQuant to improve low-bit quantization without explicit runtime permutation operations. Extensive experiments on multiple large diffusion models show that PermuQuant consistently reduces quantization error and delivers strong performance under extremely low-bit settings. We further validate its practical efficiency on Flux.1-dev with an RTX 5090,