DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion

Yongji Long, Shijun Liang, Jintao Li, Yun Li · 2026

Read DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion by Yongji Long, Shijun Liang, Jintao Li, Yun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20470.txt.

Introduction

Long, Liang, Li, and Li present DynamicRad, a sparse-attention method that speeds video generation by skipping most attention links. It pairs a fixed-ratio mode for speed with a threshold mode for quality, both guided by a radial locality prior, a bias toward nearby frames. An offline tuning step maps prompts to sparsity settings without live search. On HunyuanVideo and Wan2.1-14B it runs up to 2.5 times faster at over 80 percent sparsity, and in some long clips it even matches or tops the dense baseline.

Catalog record

CREATORYongji Long, Shijun Liang, Jintao Li, Yun Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53106
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20470

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion Yongji Long, Shijun Liang, Jintao Li, Yun Li Abstract Leveraging the natural spatiotemporal energy decay in video diffusion offers a path to efficiency, yet relying solely on rigid static masks risks losing critical long-range information in complex dynamics. To address this issue, we propose DynamicRad , a unified sparse-attention paradigm that grounds adaptive selection within a radial locality prior. DynamicRad introduces a dual-mode strategy: static-ratio for speed-optimized execution and dynamic-threshold for quality-first filtering. To ensure robustness without online search overhead, we integrate an offline Bayesian Optimization (BO) pipeline coupled with a semantic motion router . This lightweight projection module maps prompt embeddings to optimal sparsity regimes with minimal runtime overhead . Unlike online profiling methods, our offline BO optimizes attention reconstruction error (MSE) on a physics-based proxy task, ensuring rapid convergence. Experiments on HunyuanVideo and Wan2.1-14B demonstrate that DynamicRad pushes the efficiency–quality Pareto frontier, achieving 1.7 × \times –2.5 × \times inference speedups with over 80% effective sparsity . In some long-sequence settings, the dynamic mode even matches or exceeds the dense baseline, while mask-aware LoRA further improves long-horizon coherence. Code is available at https://github.com/Adamlong3/DynamicRad . 1 Introduction Recent diffusion-based video generators achieve strong visual fidelity and temporal coherence, yet scaling them to long horizons and high resolution remains difficult because self-attention grows quadratically with the spatiotemporal token count ( 12 ; 30 ; 26 ; 25 ; 28 ; 13 ; 1 ; 24 ; 33 ) . Even with optimized kernels, attention-side compute and memory bandwidth remain major bottlenecks ( 7 ; 8 ; 3 ; 22 ) . A natural remedy is sparsity. Prior efficient attention methods mainly fall into two categories: static sparse patterns, which are hardware-friendly but rigid, and dynamic sparsification, which is adaptive but often introduces extra overhead or instability under heterogeneous and long-video settings ( 5 ; 2 ; 41 ; 16 ; 35 ; 6 ; 27 ; 32 ) . Recent evidence further suggests a spatiotemporal locality structure in video diffusion attention ( 19 ; 43 ; 9 ) , where post-softmax attention mass decays with spatial and temporal distance, motivating sparse masks that preserve near-field interactions while pruning long-range pairs ( 19 ; 43 ) . However, purely static sparsification remains brittle under complex motion, whereas fully dynamic schemes may be costly or unstable at scale. In this work, we propose DynamicRad , a unified sparse-attention framework for long video diffusion that combines structured sparsity with content adaptivity. DynamicRad uses a shared block-sparse candidate set with two modes: static-ratio for throughput-first deployment and dynamic-threshold for quality-first filtering. To avoid online search overhead, we pair this design with offline BO-based configuration, a lightweight semantic motion router, and an optional mask-aware LoRA for robustness under aggressive sparsity. Contributions We summarize our contributions as follows: (1) Core Framework: We propose DynamicRad, a unified sparse-attention framework that combines a shared kernel-friendly candidate set with two selection modes, bridging rigid-prior methods such as Radial Attention ( 19 ) and higher-overhead dynamic methods such as SVG ( 38 ) and STA ( 43 ) . We further introduce a multi-dimensional parameterization, including a continuous long-range factor \lambda , that unifies static efficiency and dynamic flexibility without auxiliary networks. (2) Robust Deployment: We introduce an offline BO-driven configuration pipeline coupled with a lightweight Semantic Motion Router . By optimizing Attention MSE on a physics-based proxy task , the search process can be completed in minutes, while the router maps prompt embeddings to optimal regimes in <2 ms without fragile manual tuning. (3) Empirical Gains: Integrating DynamicRad into HunyuanVideo-241f and Wan2.1-14B ( 23 ; 18 ; 34 ; 15 ) , we achieve up to 2.54 \times speedup and show that the dynamic mode can even outperform dense attention in quality under long sequences. A mask-aware LoRA add-on further improves temporal coherence under aggressive sparsity. 2 Related Work Evolution of Video Generation Backbones. Video generation has shifted from U-Net backbones ( 28 ) to DiT-style architectures ( 25 ) due to superior scalability. Representative models such as Latte ( 21 ) , CogVideoX ( 40 ) , VideoPoet ( 17 ) , and long-context systems ( 11 ) extend generation length and quality, but also amplify the attention bottleneck. Sparse Attention & Efficiency Paradigms. Scaling diffusion models to long horizons is hindered by the quadratic \mathcal{O}(N^{2}) cost of attention ( 33 ; 7 ) . Prior works typically bifurcate into static patterns that impose fixed locality priors ( 5 ; 19 ) and dynamic routing that adapts to content ( 27 ; 38 ; 43 ) . While Radial Attention ( 19 ) is hardware-friendly, it lacks adaptivity. Conversely, SVG ( 38 ) enables dynamic routing but adds online overhead. To address these gaps, recent works have introduced novel structural efficiencies. Dynamic Diffusion Transformer (DyDiT) ( 45 ) dynamically adjusts computation in diffusion transformers along timestep and spatial dimensions, while training-free baselines such as PowerAttention (PA) ( 4 ) , SVG ( 38 ) , and Radial Attention ( 19 ) improve efficiency from different design perspectives. Learning-based Efficiency. Recent approaches like VORTA ( 31 ) and VSA ( 42 ) propose learning sparsity policies directly via gating networks. While effective, they require expensive retraining or per-step routing inference. In contrast, DynamicRad achieves adaptivity in a training-free manner (for the attention mechanism itself) by leveraging physical motion priors and reusing pre-computed text embeddings, avoiding the need for invasive architectural changes. Deployment Optimization Strategies. Our work integrates several orthogonal efficiency techniques. Similar to TeaCache ( 20 ) and DeepCache ( 22 ) which skip layers to save compute, we reduce the intra-layer attention cost. To ensure temporal coherence under aggressive pruning, we employ Parameter-Efficient Fine-Tuning (PEFT) akin to LoRA ( 14 ) , but specifically target projection layers sensitive to sparsity. Finally, instead of costly online neural architecture search (NAS) ( 29 ) , we utilize an offline Bayesian Optimization pipeline to pre-compile a lightweight lookup table, enabling instantaneous adaptation at runtime. 3 Method Figure 1: Overview of DynamicRad. Offline BO builds a lookup table of motion-regime-specific sparsity configurations. At inference time, a semantic router selects a regime from the prompt embedding, which determines either static-ratio or dynamic-threshold selection within a shared structured candidate set to produce the final block-sparse mask. An optional mask-aware LoRA compensates for sparsity-induced information loss. Radial Attention reduces attention cost via a static energy-decay mask ( 19 ) , but its fixed sparsity pattern can be brittle under long horizons and complex motion where long-range dependencies become critical. To resolve these limitations, we propose DynamicRad , a unified sparse-attention paradigm summarized in Figure 1 . The framework combines offline BO-based configuration, prompt-conditioned regime routing, and a shared candidate set with two selection modes: static-ratio for throughput-first deployment and dynamic-threshold for quality-first deployment. An optional mask-aware LoRA further compensates for sparsity-induced degradation. Both modes share the same kernel-friendly block aggregation logic, and the following subsections present the formulation. 3.1 Dynamic Neighborhood Partitioning and Parameter Space To ground the decisions of our Semantic Router within a mathematically rigorous framework, we formalize a multi-dimensional parameter space that governs the attention connectivity. Rather than treating these as static hyperparameters, we

Read the complete text