MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation

Weilun Feng, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Boyu Diao, Fuzhen Zhuang, Michele Magno, Yongjun Xu, Yingli Tian, Tingwen Huang · 2025

Read MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation by Weilun Feng, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Boyu Diao, Fuzhen Zhuang, Michele Magno, Yongjun Xu, Yingli Tian, Tingwen Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04290.txt.

Introduction

Weilun Feng and co-authors present MPQ-DMv2 in a 2025 paper on shrinking diffusion models for edge devices. Such models generate strong images but cost much compute. Quantization, storing numbers in fewer bits, can speed them and save memory, yet very low bit settings break existing methods.

The team identify problems in quantizer design, starting values, and training strategy. Their framework adds a residual branch that handles outlier values with flexible steps. A temporal distillation idea guides training across time steps. Analysis of a common adapter module supports the design, aiming at two to four bit models that still perform.

Catalog record

CREATORWeilun Feng, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Boyu Diao, Fuzhen Zhuang, Michele Magno, Yongjun Xu, Yingli Tian, Tingwen Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation Weilun Feng, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Boyu Diao, Fuzhen Zhuang, Michele Magno, Yongjun Xu, Yingli Tian, Tingwen Huang Abstract Diffusion models have demonstrated remarkable performance on vision generation tasks. However, the high computational complexity hinders its wide application on edge devices. Quantization has emerged as a promising technique for inference acceleration and memory reduction. However, existing quantization methods do not generalize well under extremely low-bit (2-4 bit) quantization. Directly applying these methods will cause severe performance degradation. We identify that the existing quantization framework suffers from the outlier-unfriendly quantizer design, suboptimal initialization, and optimization strategy. We present MPQ-DMv2, an improved M ixed P recision Q uantization framework for extremely low-bit D iffusion M odels. For the quantization perspective, the imbalanced distribution caused by salient outliers is quantization-unfriendly for uniform quantizer. We propose Flexible Z-Order Residual Mixed Quantization that utilizes an efficient binary residual branch for flexible quant steps to handle salient error. For the optimization framework, we theoretically analyzed the convergence and optimality of the LoRA module and propose Object-Oriented Low-Rank Initialization to use prior quantization error for informative initialization. We then propose Memory-based Temporal Relation Distillation to construct an online time-aware pixel queue for long-term denoising temporal information distillation, which ensures the overall temporal consistency between quantized and full-precision model. Comprehensive experiments on various generation tasks show that our MPQ-DMv2 surpasses current SOTA methods by a great margin on different architectures, especially under extremely low-bit widths. I Introduction Diffusion Models (DMs) [ 1 , 2 ] have recently emerged as a powerful generative paradigm, demonstrating superior performance across a wide range of vision tasks, including image generation [ 3 , 4 ] , video synthesis [ 5 , 6 , 7 ] , image restoration [ 8 , 9 ] , and audio generation [ 10 ] . The success largely stems from the iterative denoising mechanism of diffusion models, which denoise the generation target gradually from a Gaussian noise. This enables diffusion models to model complex data distributions with high fidelity. However, such iterative sampling also imposes significant computational burdens during inference, particularly when dealing with high-resolution data, as each forward pass must process a massive number of parameters repeatedly [ 11 ] . This poses a major obstacle to deploying diffusion models in latency-sensitive or resource-constrained scenarios, such as mobile devices and edge computing environments [ 12 , 13 , 14 ] . Fig. 1: The FID score for LDM-8 LSUN-Churches model under different quantization settings, lower FID indicates better performance. WxAy denotes x-bit weight and y-bit activation quantization, e.g., W2A4 denotes 2-bit weight and 4-bit activation quantization. Our MPQ-DMv2 surpasses current quantization methods by a great margin. To address the computational inefficiency of diffusion models, quantization has been introduced as an effective compression technique. By mapping full-precision floating-point representations to low-bit integers, quantization significantly reduces memory footprint and inference latency [ 15 ] . It has been extensively adopted in both convolutional network [ 16 , 17 , 18 ] , Transformer [ 19 ] , and Mamba [ 20 , 21 ] . Despite these successes, quantizing diffusion models remains highly challenging due to their unique denoising dynamics and the sequential accumulation of discretization errors over denoising time steps. Unlike deterministic models, the intermediate representations in diffusion models are more sensitive to quantization perturbation. Small quantization errors can be amplified across time steps, leading to quality degradation in the final output. To mitigate this issue, quantization-aware training (QAT) [ 22 , 23 , 24 ] has been explored in diffusion models [ 25 , 26 , 27 ] , allowing fine-tuning of weights and quantization parameters using large training data. While QAT often yields strong performance, especially under low-bit settings or even binarization, it requires expensive re-training resources comparable to the original model training. In contrast, post-training quantization (PTQ) [ 28 , 29 , 30 , 31 ] provides a lightweight alternative by calibrating quantization parameters with a small subset of calibration data. This makes it possible to quantize the pre-trained models while utilizing a small amount of computing resources and maintaining high precision. However, existing PTQ-based methods typically struggle under extremely low bit-widths (e.g., 2-4 bits) due to the limited adaptation capacity. Fig. 2: Overview of proposed MPQ-DM2 framework. The framework consists of Flexible Z-Order Residual Mixed Quantization to use efficient binary branch for flexible quantizer design, Memory-based Temporal Relation Distillation for denoising temporal consistency distillation, and Object-Oriented Low-Rank Initialization to use prior quantization error for informative initialization. To overcome the performance degradation, Quantization-Aware Low-Rank Adaptation (QA-LoRA) scheme [ 32 , 33 ] is introduced into diffusion models. QA-LoRA uses LoRA [ 34 ] technique to perform low-rank updates on the quantization weights, which does not affect the quantization process and makes the weight distribution more suitable for quantization operations, thereby improving the quantization performance. QA-LoRA can even achieve QAT-level performance under PTQ-like calibration budgets. While this approach significantly improves robustness, our empirical study reveals that it still suffers from noticeable performance degradation in extremely low-bit settings (e.g., 2-3 bits). We attribute this to three critical bottlenecks in existing quantization pipelines for diffusion models: (1) Inflexible quantizer design. Existing quantization frameworks [ 32 , 33 ] adopt uniform quantization strategies for the model weights. However, we observe significant variance in channel-wise weight distributions, including sparse and salient outliers. Uniform quantizers fail to effectively allocate bit-widths for such heterogeneous statistics, leading to suboptimal encoding of critical weights. (2) Isolated temporal supervision. Existing methods [ 32 , 33 ] use time-wise activation quantization strategy to utilize different quantization parameters for different denoising steps. While this strategy ensures parameter adaptation to variant activation distribution, it optimizes intermediate features independently at each timestep, ignoring the strong temporal correlation across denoising stages. This results in temporally inconsistent latent trajectories, which lack the perception of the overall denoising trajectory and result in significant deviation of the final denoising result. (3) Cold-start low-rank adaptation. QA-LoRA modules are often initialized with zero matrix statistics, neglecting the value prior information of the quantization error. This causes a cold-start optimization scenario where the module must learn corrections from scratch, thereby resulting in larger quantization error and slowing the optimization convergence. To address the quantization limitations of extremely low-bit quantization, we propose a unified quantization framework, Flexible Residual M ixed P recision Q uantization for Low-Bit D iffusion M odels with Temporal Distillation (MPQ-DMv2). In Fig. 1 , we compare our proposed MPQ-DMv2 with current SOTA methods. MPQ-DMv2 significantly surpasses existing methods by a large margin. The overview of the proposed framework is in Fig. 2 . The proposed MPQ-DMv2 framework mainly relies on three novel techniques: Flexible Z-Order Residual Mixed Quantization (FZ-RMQ): utilizes a primary quantizer for the central weight distribution and a lightweight residual quantizer that uses binary codes to capture salient residuals. This residual pathway effectively preserves important outliers with minimal overhead. Memory-based Temporal Relation Distillation (MTRD): introduces a memory-based online distillation mechanism that aligns temporal consistency by leveraging long-term structural dependencies. Object-Oriented Low-Rank Initialization (OOLRI): initializes the low-rank module by approximating the prior quantization error using truncated singular value decomposition.

Read the complete text