Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers

Weilun Feng, Chuanguang Yang 🖂, Haotong Qin, Xiangqi Li, Yu Wang, Zhulin An 🖂, Libo Huang, Boyu Diao, Zixiang Zhao, Yongjun Xu, Michele Magno · 2025

Read Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers by Weilun Feng, Chuanguang Yang 🖂, Haotong Qin, Xiangqi Li, Yu Wang, Zhulin An 🖂, Libo Huang, Boyu Diao, Zixiang Zhao, Yongjun Xu, Michele Magno (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22167.txt.

Introduction

Weilun Feng and co-authors shrink video diffusion transformers with a framework called Q-VDiT. These models make striking video but carry huge parameter counts that bar them from edge devices. Quantization cuts the bit width of the weights, yet image-born methods bend when frames must stay in step.

Their 2025 paper names two faults: information lost in the squeeze, and training aims that ignore video's own needs. A token-aware estimator pays back quantization error along both token and feature axes. A temporal distillation step then guards the links between frames while each frame is tuned in full context. At very low bit width, their model sets a fresh scene-consistency mark.

Catalog record

CREATORWeilun Feng, Chuanguang Yang 🖂, Haotong Qin, Xiangqi Li, Yu Wang, Zhulin An 🖂, Libo Huang, Boyu Diao, Zixiang Zhao, Yongjun Xu, Michele Magno
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers Weilun Feng, Chuanguang Yang 🖂, Haotong Qin, Xiangqi Li, Yu Wang, Zhulin An 🖂, Libo Huang, Boyu Diao, Zixiang Zhao, Yongjun Xu, Michele Magno Abstract Diffusion transformers (DiT) have demonstrated exceptional performance in video generation. However, their large number of parameters and high computational complexity limit their deployment on edge devices. Quantization can reduce storage requirements and accelerate inference by lowering the bit-width of model parameters. Yet, existing quantization methods for image generation models do not generalize well to video generation tasks. We identify two primary challenges: the loss of information during quantization and the misalignment between optimization objectives and the unique requirements of video generation. To address these challenges, we present Q-VDiT , a quantization framework specifically designed for video DiT models. From the quantization perspective, we propose the Token-aware Quantization Estimator (TQE), which compensates for quantization errors in both the token and feature dimensions. From the optimization perspective, we introduce Temporal Maintenance Distillation (TMD), which preserves the spatiotemporal correlations between frames and enables the optimization of each frame with respect to the overall video context. Our W3A6 Q-VDiT achieves a scene consistency of 23.40, setting a new benchmark and outperforming current state-of-the-art quantization methods by 1.9 × \times . Code will be available at https://github.com/cantbebetter2/Q-VDiT . 1 Introduction Figure 1: Evaluation on VBench of different quantization methods under W3A6 setting. Figure 2: Overview of proposed Q-VDiT. The framework includes Token-aware Quantization Estimator (TQE) for forward process and Temporal Maintenance Distillation (TMD) for optimization. The middle part denotes the quantized forward process. \otimes denotes matrix multiplication, \odot denotes token-wise multiplication. Diffusion models (DMs) ( Ho et al., 2020 ; Song et al., 2020 ) have demonstrated remarkable success across a wide range of generative tasks, including image generation ( Ho et al., 2020 ; Rombach et al., 2022 ; Dhariwal & Nichol, 2021 ; Feng et al., 2024 ; Yang et al., 2025 ) , image super-resolution ( Lin et al., 2025 ; Wang et al., 2024b ; Wu et al., 2024b ) , and video generation ( HPC-AI, 2024 ; Ma et al., 2024 ) . Diffusion Transformers (DiT) ( Peebles & Xie, 2023 ) have emerged as a prominent architecture for generation tasks, leveraging their ability to capture long-range dependencies and scale to large parameter spaces. However, up to billions of parameters ( Black-Forest-Labs, 2024 ) and the increased computational complexity pose significant challenges for their deployment on edge devices. As an effective model compression technique, quantization reduces the bit-width of parameters and enhances inference speed by utilizing integer operations ( Gholami et al., 2022 ) . This approach has been extensively applied to compress CNN-based ( Pilipović et al., 2018 ; Ding et al., 2024 ) and Transformer-based ( Chitty-Venkata et al., 2023 ) architectures. However, existing quantization methods for Diffusion Transformers (DiT) have primarily focused on image generation tasks, with limited exploration in the context of video generation. Directly applying existing quantization methods ( Li et al., 2023 ; Ashkboos et al., 2024 ; Chen et al., 2024 ; Wu et al., 2024a ) to video-generation DiT leads to significant performance degradation ( Zhao et al., 2024 ) . Compared to image generation, video generation models require modeling additional temporal dimensions across multiple frames, significantly increasing the information density relative to image generation tasks. Quantization introduces information loss ( Qin et al., 2020 ; Liu et al., 2020 ; Feng et al., 2025 ) , which can lead to considerable performance degradation in video generation models, given their higher information density ( Zhao et al., 2024 ) . Furthermore, video generation involves strong semantic and temporal correlations between frames. Existing optimization methods ( Wu et al., 2024a ; He et al., 2023 ) typically use Mean Squared Error (MSE) to align model outputs directly, which fails to account for these correlations. As a result, these approaches do not calibrate the quantization process from the perspective of the entire video, leading to a degradation in video quality. To address the aforementioned challenges, we propose Q-VDiT ( Q uantization of V ideo-Generation Di ffusion T ransformers), a novel framework specifically designed for quantizing video diffusion transformers. Q-VDiT comprises two key components: Token-aware Quantization Estimator (TQE) and Temporal Maintenance Distillation (TMD). An overview of Q-VDiT is illustrated in Fig. 2 . From quantization perspective, quantizing weights introduces significant information loss, which is a primary factor contributing to the degradation of video quality. To mitigate this, we employ a small number of parameters to estimate the low-rank quantization error of the weights from two orthogonal tokens and feature dimensions. From optimization perspective, relying solely on Mean Squared Error (MSE) for optimization neglects the inter-frame information and fails to capture the overall temporal dynamics of the video. We construct the similarity distribution between frames in the full-precision (FP) model as prior knowledge. Using KL divergence, we align the temporal distributions of the quantized model with the FP model, enabling the quantized model to maintain semantic and temporal coherence across frames. We compare the quantized model’s performance in Fig. 1 . The main contributions are summarized as follows: • We theoretically prove that the quantization error of weights carries less information entropy than the original weights. To address this, we introduce the Token-aware Quantization Estimator (TQE), which employs a small number of additional parameters to perform low-rank approximation of the quantization errors from two orthogonal token and feature dimensions. • We identify that using MSE alone fails to capture the inter-frame optimization information in video generation. To address this, we propose Temporal Maintenance Distillation (TMD), which models the inter-frame distribution to ensure that the optimization of each frame considers the overall distribution of video characteristics across all frames. • We propose Q-VDiT, a framework specifically designed for quantizing video diffusion transformers. Extensive experiments on generative benchmarks show that Q-VDiT significantly outperforms current SOTA post-training quantization methods. 2 Related Work 2.1 Diffusion Model Diffusion models ( Ho et al., 2020 ; Rombach et al., 2022 ) perform a forward sampling process by gradually adding noise to the data distribution \mathbf{x}_{0}\sim q(x) . In DDPM, the forward noise addition process of the diffusion model is a Markov chain, taking the form: \begin{gathered}q(\mathbf{x}_{1:T}|\mathbf{x}_{0})=\prod\limits_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}),\\ q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\alpha_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}),\end{gathered} (1) where \alpha_{t}=1-\beta_{t} , \beta_{t} is time-related schedule. Diffusion models generate high-quality images by applying a denoising process to randomly sampled Gaussian noise \mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) , taking the form: p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\hat{\mu}_{\theta,t}(\mathbf{x_{t}}),\hat{\beta}_{t}\mathbf{I}), (2) where \hat{\mu}_{\theta,t} and \hat{\beta}_{t} are outputed by the diffusion model. 2.2 Diffusion Quantization For diffusion model quantization, methods such as Q-DM ( Li et al., 2024b ) , BinaryDM ( Zheng et al., 2024b ) , BiDM ( Zheng et al., 2024a ) , and TerDiT ( Lu et al., 2024 ) use quantization-aware training to maintain model performance under 1-2 bits. However, these approaches require extensive additional training time, often lasting several days. For more efficient quantization, approaches like Q-Diffusion ( Li et al., 2023 ) , PTQ4DM ( Shang et al., 2023 ) , PTQ-D ( He et al., 2024 ) , TFMQ-DM ( Huang et al., 2024a ) , QuEST ( Wang et al., 2024a ) , EfficientDM ( He et al., 2023 ) , and MixDQ

Read the complete text