S 2 \text{S}^{2} Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation

Weilun Feng, Haotong Qin, Chuanguang Yang, Xiangqi Li, Han Yang, Yuqi Li, Zhulin An, Libo Huang, Michele Magno, Yongjun Xu · 2025

Read S 2 \text{S}^{2} Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation by Weilun Feng, Haotong Qin, Chuanguang Yang, Xiangqi Li, Han Yang, Yuqi Li, Zhulin An, Libo Huang, Michele Magno, Yongjun Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04016.txt.

Introduction

Weilun Feng and co-authors present a 2025 paper on shrinking video generation models so they run faster and cheaper. Diffusion transformers that make videos carry billions of settings and cost heavily at each step. Compressing their numbers to fewer bits saves memory, but video's long token sequences make careful compression tricky.

Their framework, S-squared Q-VDiT, tackles two weak points. It picks calibration clips using a sensitivity measure tied to both diffusion and compression. It also retrains the model with extra attention to the tokens that matter most, which video attention uses sparsely. At a tight four-bit setting the compressed model kept full quality while shrinking nearly fourfold and running about a third faster.

Catalog record

CREATORWeilun Feng, Haotong Qin, Chuanguang Yang, Xiangqi Li, Han Yang, Yuqi Li, Zhulin An, Libo Huang, Michele Magno, Yongjun Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S 2 \text{S}^{2} Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation Weilun Feng, Haotong Qin, Chuanguang Yang, Xiangqi Li, Han Yang, Yuqi Li, Zhulin An, Libo Huang, Michele Magno, Yongjun Xu Abstract Diffusion transformers have emerged as the mainstream paradigm for video generation models. However, the use of up to billions of parameters incurs significant computational costs. Quantization offers a promising solution by reducing memory usage and accelerating inference. Nonetheless, we observe that the joint modeling of spatial and temporal information in video diffusion models (V-DMs) leads to extremely long token sequences, which introduces high calibration variance and learning challenges. To address these issues, we propose S 2 \text{S}^{2} Q-VDiT , a post-training quantization framework for V-DMs that leverages S alient data and S parse token distillation. During the calibration phase, we identify that quantization performance is highly sensitive to the choice of calibration data. To mitigate this, we introduce Hessian-aware Salient Data Selection , which constructs high-quality calibration datasets by considering both diffusion and quantization characteristics unique to V-DMs. To tackle the learning challenges, we further analyze the sparse attention patterns inherent in V-DMs. Based on this observation, we propose Attention-guided Sparse Token Distillation , which exploits token-wise attention distributions to emphasize tokens that are more influential to the model’s output. Under W4A6 quantization, S 2 \text{S}^{2} Q-VDiT achieves lossless performance while delivering 3.9 × 3.9\times model compression and 1.3 × 1.3\times inference acceleration. Code will be available at https://github.com/wlfeng0509/s2q-vdit . Figure 1: We present \text{S}^{2} Q-VDiT, a post-training quantization method for video diffusion transformers. We quantize HunyuanVideo [ 19 ] to 4-bit weights and 6-bit activations without compromising visual quality. \text{S}^{2} Q-VDiT can further achieve 3.9\times model compression and 1.3\times inference acceleration. 1 Introduction In recent years, diffusion transformer [ 34 ] has emerged as a powerful generative paradigm, demonstrating remarkable performance across diverse domains such as image synthesis [ 5 , 21 ] , audio generation [ 10 ] , and increasingly, video generation [ 32 , 30 ] . Among these, video diffusion models (V-DMs) [ 51 , 19 ] represent a new frontier by extending the spatial generative capabilities of image diffusion models (I-DMs) into the spatial-temporal domain, enabling high-quality video synthesis from textual prompts. However, the transition from image to video generation introduces substantial computational challenges, primarily due to the exponential growth in token count introduced by the temporal dimension [ 30 , 51 , 19 ] . These memory and compute demands become particularly severe in large-scale video generation models [ 30 , 51 , 19 ] , which contain up to billions of parameters, where each input consists of thousands or even tens of thousands of tokens. To enable efficient deployment of such models in resource-constrained environments, post-training quantization (PTQ) [ 27 , 47 , 15 ] has become a widely adopted approach. PTQ compresses the pre-trained models into low-bit representations without modifying the model weights, relying only on a small dataset to calibrate quantization parameters with only hours on a single GPU [ 46 , 23 ] . While PTQ has proven effective for I-DMs [ 25 , 40 , 49 ] , directly applying it to V-DMs leads to substantial performance degradation [ 2 , 55 ] . Prior works [ 2 , 49 , 55 ] have sought to improve V-DMs’ quantization performance primarily from the perspective of quantizer design. In this paper, we delve deeper into the PTQ challenges specific to V-DMs, focusing on calibration data and optimization methods. We identify that the long token sequences characteristic of V-DMs significantly constrain the number of calibration samples (e.g., thousands for I-DMs vs. only dozens for V-DMs under equal computational budgets). Under such limited budgets, quantization performance becomes highly sensitive to the selection of calibration samples. Existing methods [ 49 , 2 , 55 ] typically employ random or uniform sampling strategies, which work reasonably well for I-DMs but fail to generalize well to only dozens of data for V-DMs. Moreover, we observe that V-DMs exhibit sparse attention patterns across all tokens. Current PTQ optimization frameworks [ 49 , 25 ] treat all tokens equally during loss alignment between full-precision and quantized models. However, this uniform treatment is suboptimal for long token sequences, where only a small subset of tokens significantly impacts the final output. These observations highlight two fundamental challenges in PTQ for V-DMs: (1) the absence of a principled method for selecting calibration samples, and (2) the inefficiency of uniform token treatment during optimization, despite the varying importance of tokens. To address these limitations, we propose \text{S}^{2} Q-VDiT , a post-training quantization framework tailored for V-DMs, built upon S alient data selection and S parse token distillation. An overview of the proposed framework is illustrated in Fig. 2 . First, we introduce Hessian-aware Salient Data Selection , which constructs calibration datasets by jointly assessing diffusion informativeness and quantization sensitivity. We define a unified metric to quantify sample’s saliency to the denoising process and its sensitivity to quantization perturbations. Second, we present Attention-guided Sparse Token Distillation , a technique that leverages the inherent sparsity of spatial-temporal attention in V-DMs. Rather than treating all tokens equally during optimization, we reweight quantization losses based on token-wise attention distribution, allowing the model to focus more on the impactful representations. Our main contribution can be summarized as follows: • We empirically identify that V-DMs suffer from high calibration data variance in quantization performance. We propose Hessian-aware Salient Data Selection , which jointly considers diffusion informativeness and quantization sensitivity to construct effective calibration datasets. • We introduce Attention-guided Sparse Token Distillation , a method that leverages the inherent sparsity in spatial-temporal attention of V-DMs. We reweight the quantization loss of different tokens by measuring token-wise attention distribution. This enables the model to focus more on the impactful representations during optimization. • Extensive experiments on large-scale video diffusion transformers with 2B to 13B parameters demonstrate that our \text{S}^{2} Q-VDiT consistently outperforms existing PTQ baselines, achieving state-of-the-art performance under all quantization settings. 2 Related Works Diffusion models [ 41 , 12 ] have demonstrated strong generative capabilities in video generation tasks. However, up to billions of parameters [ 30 , 44 , 51 , 19 ] pose major challenges for deployment in resource-constrained environments. Quantization has emerged as a widely adopted solution for model compression and acceleration [ 35 , 9 , 3 , 16 , 20 ] . A growing body of work has explored post-training quantization (PTQ) for diffusion models, particularly focusing on U-Net-based architectures [ 25 , 40 , 11 , 13 , 24 , 56 ] . For the Diffusion Transformer architecture specifically, recent methods [ 49 , 2 ] have made further explorations from the perspective of data distribution and architecture characteristics on quantization behavior. To address performance degradation at ultra-low bits, several quantization-aware training approaches have been proposed [ 57 , 26 , 31 , 58 ] . While effective, these methods typically require extensive training time and large-scale datasets, making them less practical in many scenarios. Despite these advances, most existing quantization research remains focused on image diffusion models (I-DMs), with limited exploration of video diffusion models (V-DMs). ViDiT-Q [ 55 ] and Q-DiT

Read the complete text