Training-Free Sparse Attention for Fast Video Generation via Offline Layer-Wise Sparsity Profiling and Online Bidirectional Co-Clustering

Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, Jianxin Li · 2026

Read Training-Free Sparse Attention for Fast Video Generation via Offline Layer-Wise Sparsity Profiling and Online Bidirectional Co-Clustering by Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, Jianxin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18636.txt.

Introduction

Luo and co-authors speed up video diffusion models by exploiting stable attention sparsity. Their SVOO framework needs no training. Offline profiling first measures how sparse each layer can safely go. Those levels barely change across inputs, the team reports. Online, a two-way co-clustering of queries and keys shapes sparse blocks. This respects both layer differences and query-key coupling. Across seven video models, SVOO reaches up to 1.93 times speedup. Quality holds near 29 dB PSNR on Wan2.1, beating prior sparse methods on the trade-off.

Catalog record

CREATORJiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, Jianxin Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77020
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training-Free Sparse Attention for Fast Video Generation via Offline Layer-Wise Sparsity Profiling and Online Bidirectional Co-Clustering Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, Jianxin Li Abstract Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, leading to the development of sparse attention technologies to improve efficiency. However, existing training-free sparse attention methods in video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning , which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight that the attention sparsity of each layer is its intrinsic property, with minor effects across different inputs . Motivated by this, we propose SVOO , a training-free S parse attention framework for fast V ideo generation via O ffline layer-wise sparsity profiling and O nline bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a novel bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to 1.93 × 1.93\times speedup while maintaining a PSNR of up to 29 dB on Wan2.1. 1 Introduction Diffusion Transformers(DiTs) ( Peebles & Xie 2023 ) have already revolutionized the video generation, achieving high fidelity and temporal coherence. Despite their success ( Wan et al. 2025 ; Yang et al. 2024 ; Team et al. 2025 ; Zheng et al. 2024 ) , DiTs incur prohibitive computational overhead, primarily due to their dense 3D self-attention with quadratic complexity in spatial-temporal token count ( Zhang et al. 2025b ; Contributors 2025 ; Team 2024 ; Lin et al. 2024 ) . Figure 1 : An example acceleration comparison on the Wan2.1-T2V-1.3B ( Wan et al. 2025 ) model against a state-of-the-art training-free sparse attention method named SVG2 ( Yang et al. 2025 ) . Our proposed SVOO achieves higher inference speedup (1.96 \times vs. 1.75 \times ) while still maintaining high generation quality in practice . All experiments are conducted on a single NVIDIA H200 GPU at a 720p resolution of 720 \times 1280 with 81 frames. Recent studies mitigate the high computational cost of DiT attention by exploiting redundancy in attention mechanisms, motivated by empirical evidence that attention maps are highly sparse, with only a small fraction of attention weights being non-negligible ( Zhang et al. 2025c ; Sun et al. 2025 ) . This observation has led to a series of sparse attention methods, which can be broadly divided into training-free approaches ( Zhang et al. 2025a ; Shen et al. 2025 ; Yang et al. 2025 ; Xi et al. 2025 ; Xu et al. 2025 ; Li et al. 2025 ) and training-based approaches ( Wu et al. 2025 ; Zhang et al. 2025e ; Tan et al. 2025 ; Zhan et al. 2025 ) , where training-free sparse attention reduces computation by directly leveraging sparsity without incurring additional training cost. In practice, training-free sparse attention is often implemented with a coarse-to-fine pipeline: tokens are partitioned into blocks, block importance is estimated efficiently, and the dense attention is computed only for a subset of block pairs. However, while these training-free sparse attention methods substantially reduce the inference cost of DiT-based video generation models, they still face two unresolved limitations that hinder a better satisfying speed-quality trade-off: • L1: Ignore Layer Heterogeneity in Attention Pruning : Most existing methods treat the multiple transformer layers as a homogeneous stack and apply uniform sparsity ratios across layers. In Sec. 3 and 4.1 , we empirically and theoretically show that attention sparsity is an intrinsic property of each layer, exhibiting pronounced variation across layers while remaining relatively stable within each layer across different inputs . Such existing layer-agnostic designs overlook the distinct functional roles of different layers and their varying tolerance to attention pruning, leading to suboptimal sparsification decisions. • L2: Ignore Q-K Coupling in Block Partitioning : Existing block-wise sparse attention methods partition queries and keys into blocks independently, despite the fact that salient attention patterns emerge from coupled Q-K interactions. In Sec. 3 , our analysis indicates that the optimal block partitioning of keys is query-dependent, and vice versa . The existing Q-K decoupled blocking may misalign Q-K informative correspondences, leading to inferior sparsity patterns and reduced generation fidelity. To address these challenges, we propose SVOO , a training-free S parse attention framework for fast V ideo generation via O ffline layer-wise sparsity profiling and O nline bidirectional co-clustering. Specifically, we first quantify the intrinsic pruning tolerance of each transformer layer via an offline calibration on a small set of random inputs, and derive a sparsity schedule that specifies the appropriate pruning ratio per layer. During inference, this schedule is applied to guide attention sparsification, delivering speedups with minor impact on quality. Next, to efficiently obtain a coupling-aware block partition without dense computation, we introduce a bidirectional co-clustering scheme that jointly groups queries and keys. Tokens are assigned according to their cross-attention affinity to opposite-side centroids and iteratively refined, producing well-aligned Q-K blocks with negligible overhead. We evaluate SVOO on 7 widely used models, including Wan2.1-T2V-1.3B, Wan2.1-T2V-14B, Wan2.1-I2V-14B, Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, HunyuanVideo-T2V, and HunyuanVideo-I2V. SVOO consistently achieves a better trade-off between quality and efficiency than the state-of-the-art training-free sparse attention methods. Our contributions are summarized as follows: • We conduct an in-depth analysis of existing training-free sparse attention methods and reveal two unresolved limitations: ignoring layer heterogeneity in attention pruning and overlooking Q-K coupling in block partitioning. • We propose SVOO, a novel training-free sparse attention framework tailored for fast video generation via offline layer-wise sparsity profiling and online bidirectional co-clustering to solve the aforementioned two limitations. • Extensive experiments across 7 widely used video generation models demonstrate that SVOO offers a better trade-off between generation quality and inference speedup. 2 Related Work Existing sparse attention methods for fast video generation can be broadly categorized into training-based sparse attention and training-free sparse attention. In this work, our method focuses on training-free sparse attention methods. 2.1 Training-based Sparse Attention A recent line of work focuses on trainable sparse attention, where sparsity patterns are learned during training. VMoBA ( Wu et al. 2025 ) accelerates video DiTs by employing a layer-wise recurrent 1D-2D-3D block partitioning scheme and a global threshold-based selection strategy. VSA ( Zhang et al. 2025d ) learns an end-to-end block-sparse attention using a coarse-to-fine tile selection scheme with annealed dense-to-sparse training. DSV ( Tan et al. 2025 ) trains per-module predictors to approximate attention scores and pre-select critical key-value pairs, enabling fused sparse attention and sparsity-aware parallelism for faster training. BSA ( Zhan et al. 2025 ) jointly sparsifies queries and key-value blocks via semantic query selection and dynamic KV thresholding under an annealed sparsity schedule to accelerate training and inference. While their promising speedups, training-based methods introduce extra training overhead. 2.2 Traning-free Sparse Attenion Training-free methods reduce inference computation by directly exploiting attention sparsity without introducing additional training cost. STA ( Zhang et al. 2025e ) introduces a hardware-friendly sliding-tile attention mechanism that

Read the complete text