DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan · 2026

Read DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation by Jie Hu, Zixiang Gao, Yutong He, Kun Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23445.txt.

Introduction

Hu and three colleagues speed video generation from diffusion transformers, whose full 3D attention grows quadratically with input size. Block-sparse attention helps but degrades at high sparsity, since attention maps in these models are dynamic and fine-grained. Their training-free DFSAttn framework reorders tokens along a Hilbert curve for fine sparsity with fast GPU execution, scores blocks hierarchically for importance, and caches sparse masks at adaptive ratios. It beats prior methods under heavy sparsity, reaching up to 2.1 times end-to-end speedup with quality intact.

Catalog record

CREATORJie Hu, Zixiang Gao, Yutong He, Kun Yuan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50642
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation Jie Hu, Zixiang Gao, Yutong He, Kun Yuan Abstract Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve–based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1 × \times end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at this link . 1 Introduction Diffusion Transformers (DiTs) ( 19 ) have achieved remarkable success in various applications. Recent state-of-the-art video DiTs ( 32 ; 41 ; 10 ; 26 ) demonstrate impressive capability in synthesizing high-fidelity videos by adopting spatiotemporal 3D full attention. Despite its effectiveness, 3D full attention incurs prohibitive computational overhead due to the quadratic computational complexity of attention ( 25 ) . For example, generating a 129-frame 720p video with HunyuanVideo ( 10 ) requires approximately 30 minutes on an NVIDIA H100 PCIe GPU, severely limiting practical deployment. Fortunately, attention maps exhibit inherent sparsity ( 37 ; 29 ) , where only a small subset of critical tokens dominate the output. Sparse attention methods exploit this property by constructing sparse masks to omit redundant computations. In particular, block sparse attention, which skips computations at the block level, aligns naturally with modern hardware and efficient kernels such as FlashAttention ( 2 ) , enabling high-performance attention computation. Figure 1 : Overview of DFSAttn: a) Video tokens are reordered using a 3D Hilbert curve, and sub-block attention scores are aggregated to estimate block-wise importance. The resulting masks are applied via sparse Flash Attention, while cross-attention remains dense to preserve text–video alignment. b) Sparse masks are cached and reused across diffusion timesteps. c) Structured block sparsity in the reordered sequence induces fine-grained sparsity of original spatiotemporal space. Existing training-free sparse attention methods for DiTs can be broadly categorized into static and dynamic approaches, depending on whether the block sparse mask is fixed offline or constructed on-the-fly during inference. Static methods ( 33 ; 27 ; 35 ; 1 ; 38 ; 13 ) rely on empirically designed mask patterns, while dynamic methods ( 34 ; 30 ; 36 ; 28 ; 22 ; 31 ; 15 ; 17 ) aim to identify important blocks during inference. However, methods from both paradigms suffer from severe quality degradation under high sparsity. We argue that this limitation stems from a fundamental mismatch between GPU-efficient block-wise attention operations and the distinctive attention characteristics of DiTs, which are not adequately captured by existing designs. Our first key observation is that attention maps in DiTs exhibit a dynamic and fine-grained sparsity pattern. As illustrated in Figure 2 , attention patterns vary significantly across layers and heads, with salient interactions scattered throughout the attention map. Consequently, static or coarse-grained sparse masks inevitably discard critical dependencies and lead to accuracy loss. Moreover, we observe that block sparse attention becomes progressively more effective as the diffusion process advances ( Figure 3 ), indicating that the effectiveness of sparsification is closely tied to the evolving statistical structure of the attention score matrix. To systematically investigate the underlying factors of effective block sparse attention, we develop a token representation model and derive a theoretical lower bound on attention recall, which characterizes the fraction of important attention interactions preserved after sparsification. Our analysis identifies three key factors governing sparsification quality: the sparsity budget, the inter-block similarity gap, and the block-level semantic diversity. Motivated by these insights, we introduce DFSAttn, a training-free dynamic fine-grained sparse attention method for accelerating video generation. DFSAttn incorporates three core designs, each directly addressing one of the identified factors. First, DFSAttn reorders video tokens using a 3D Hilbert curve, implicitly inducing fine-grained sparsity while aligning with GPU execution. This reordering preserves spatiotemporal locality in the 1D token sequence, thereby enlarging the inter-block similarity gap. Second, DFSAttn introduces a hierarchical block scoring mechanism that refines block importance estimation through sub-block aggregation, alleviating inaccuracies caused by mixed semantics within a block. Finally, DFSAttn adaptively reallocates the sparsity budget across diffusion steps and employs sparse mask caching, enabling a favorable balance between generation quality and efficiency. We evaluate DFSAttn on state-of-the-art video diffusion models: HunyuanVideo ( 10 ) , and Wan2.1 ( 26 ) . Experimental results show that DFSAttn consistently outperforms existing work in generation quality under high sparsity. Specifically, DFSAttn achieves an end-to-end speedup of 2.1 \times at an 80% sparsity ratio on HunyuanVideo, while maintaining high visual fidelity, reaching PSNR up to 29.38. Our contributions are as follows: • We identify the mismatch between block-wise sparse attention and DiTs’ distinctive attention patterns, and derive a theoretical lower bound on attention recall to characterize key factors governing effective block sparse attention. • We introduce DFSAttn, a training-free sparse attention framework that enables dynamic and fine-grained sparsification via token reordering, hierarchical block scoring, and adaptive sparse mask caching. • DFSAttn significantly improves video generation quality under high sparsity and delivers substantial end-to-end speedups across video diffusion models. 2 Related Work 2.1 Efficient video generation Numerous techniques have been developed to accelerate diffusion models. One major direction reduces inference cost by decreasing the number of sampling steps, through improved solvers or schedules such as DDIM ( 23 ) and DPM-Solver ( 18 ) , or via distillation and consistency-based models ( 21 ; 24 ) that approximate the full diffusion trajectory with few steps. System-level approaches, such as DistriFusion ( 12 ) , parallelize diffusion inference across multiple devices. Another line of work exploits inter-timestep redundancy by caching and reusing intermediate activations or attention results, including PAB ( 39 ) , TeaCache ( 16 ) , and AdaCache ( 9 ) . These caching-based approaches reduce redundant computation across diffusion steps but do not modify the attention computation within each step. DFSAttn is orthogonal to these methods and can be combined with them for further acceleration. 2.2 Sparse attention Sparse attention has been widely studied in large language models to alleviate the quadratic complexity of attention ( 37 ; 29 ; 8 ; 30 ; 11 ; 34 ) . More recently, sparse attention mechanisms have been adapted to DiTs for video generation, typically operating at the block level for GPU efficiency. Existing methods can be broadly categorized into static and dynamic approaches. Static methods predefine sparsity patterns offline, such as spatial-temporal masks ( 27 ) , 3D sliding windows ( 35 ) , energy-decay-based radial

Read the complete text