DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion
Yongji Long, Shijun Liang, Jintao Li, Yun Li · 2026
Read DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion by Yongji Long, Shijun Liang, Jintao Li, Yun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20470.txt.
Introduction
Long, Liang, Li, and Li present DynamicRad, a sparse-attention method that speeds video generation by skipping most attention links. It pairs a fixed-ratio mode for speed with a threshold mode for quality, both guided by a radial locality prior, a bias toward nearby frames. An offline tuning step maps prompts to sparsity settings without live search. On HunyuanVideo and Wan2.1-14B it runs up to 2.5 times faster at over 80 percent sparsity, and in some long clips it even matches or tops the dense baseline.
Catalog record
| CREATOR | Yongji Long, Shijun Liang, Jintao Li, Yun Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53106 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion Yongji Long, Shijun Liang, Jintao Li, Yun Li Abstract Leveraging the natural spatiotemporal energy decay in video diffusion offers a path to efficiency, yet relying solely on rigid static masks risks losing critical long-range information in complex dynamics. To address this issue, we propose DynamicRad , a unified sparse-attention paradigm that grounds adaptive selection within a radial locality prior. DynamicRad introduces a dual-mode strategy: static-ratio for speed-optimized execution and dynamic-threshold for quality-first filtering. To ensure robustness without online search overhead, we integrate an offline Bayesian Optimization (BO) pipeline coupled with a semantic motion router . This lightweight projection module maps prompt embeddings to optimal sparsity regimes with minimal runtime overhead . Unlike online profiling methods, our offline BO optimizes attention reconstruction error (MSE) on a physics-based proxy task, ensuring rapid convergence. Experiments on HunyuanVideo and Wan2.1-14B demonstrate that DynamicRad pushes the efficiency–quality Pareto frontier, achieving 1.7 × \times –2.5 × \times inference speedups with over 80% effective sparsity . In some long-sequence settings, the dynamic mode even matches or exceeds the dense baseline, while mask-aware LoRA further improves long-horizon coherence. Code is available at https://github.com/Adamlong3/DynamicRad . 1 Introduction Recent diffusion-based video generators achieve strong visual fidelity and temporal coherence, yet scaling them to long horizons and high resolution remains difficult because self-attention grows quadratically with the spatiotemporal token count ( 12 ; 30 ; 26 ; 25 ; 28 ; 13 ; 1 ; 24 ; 33 ) . Even with optimized kernels, attention-side compute and memory bandwidth remain major bottlenecks ( 7 ; 8 ; 3 ; 22 ) . A natural remedy is sparsity. Prior efficient attention methods mainly fall into two categories: static sparse patterns, which are hardware-friendly but rigid, and dynamic sparsification, which is adaptive but often introduces extra overhead or instability under heterogeneous and long-video settings ( 5 ; 2 ; 41 ; 16 ; 35 ; 6 ; 27 ; 32 ) . Recent evidence further suggests a spatiotemporal locality structure in video diffusion attention ( 19 ; 43 ; 9 ) , where post-softmax attention mass decays with spatial and temporal distance, motivating sparse masks that preserve near-field interactions while pruning long-range pairs ( 19 ; 43 ) . However, purely static sparsification remains brittle under complex motion, whereas fully dynamic schemes may be costly or unstable at scale. In this work, we propose DynamicRad , a unified sparse-attention framework for long video diffusion that combines structured sparsity with content adaptivity. DynamicRad uses a shared block-sparse candidate set with two modes: static-ratio for throughput-first deployment and dynamic-threshold for quality-first filtering. To avoid online search overhead, we pair this design with offline BO-based configuration, a lightweight semantic motion router, and an optional mask-aware LoRA for robustness under aggressive sparsity. Contributions We summarize our contributions as follows: (1) Core Framework: We propose DynamicRad, a unified sparse-attention framework that combines a shared kernel-friendly candidate set with two selection modes, bridging rigid-prior methods such as Radial Attention ( 19 ) and higher-overhead dynamic methods such as SVG ( 38 ) and STA ( 43 ) . We further introduce a multi-dimensional parameterization, including a continuous long-range factor \lambda , that unifies static efficiency and dynamic flexibility without auxiliary networks. (2) Robust Deployment: We introduce an offline BO-driven configuration pipeline coupled with a lightweight Semantic Motion Router . By optimizing Attention MSE on a physics-based proxy task , the search process can be completed in minutes, while the router maps prompt embeddings to optimal regimes in <2 ms without fragile manual tuning. (3) Empirical Gains: Integrating DynamicRad into HunyuanVideo-241f and Wan2.1-14B ( 23 ; 18 ; 34 ; 15 ) , we achieve up to 2.54 \times speedup and show that the dynamic mode can even outperform dense attention in quality under long sequences. A mask-aware LoRA add-on further improves temporal coherence under aggressive sparsity. 2 Related Work Evolution of Video Generation Backbones. Video generation has shifted from U-Net backbones ( 28 ) to DiT-style architectures ( 25 ) due to superior scalability. Representative models such as Latte ( 21 ) , CogVideoX ( 40 ) , VideoPoet ( 17 ) , and long-context systems ( 11 ) extend generation length and quality, but also amplify the attention bottleneck. Sparse Attention & Efficiency Paradigms. Scaling diffusion models to long horizons is hindered by the quadratic \mathcal{O}(N^{2}) cost of attention ( 33 ; 7 ) . Prior works typically bifurcate into static patterns that impose fixed locality priors ( 5 ; 19 ) and dynamic routing that adapts to content ( 27 ; 38 ; 43 ) . While Radial Attention ( 19 ) is hardware-friendly, it lacks adaptivity. Conversely, SVG ( 38 ) enables dynamic routing but adds online overhead. To address these gaps, recent works have introduced novel structural efficiencies. Dynamic Diffusion Transformer (DyDiT) ( 45 ) dynamically adjusts computation in diffusion transformers along timestep and spatial dimensions, while training-free baselines such as PowerAttention (PA) ( 4 ) , SVG ( 38 ) , and Radial Attention ( 19 ) improve efficiency from different design perspectives. Learning-based Efficiency. Recent approaches like VORTA ( 31 ) and VSA ( 42 ) propose learning sparsity policies directly via gating networks. While effective, they require expensive retraining or per-step routing inference. In contrast, DynamicRad achieves adaptivity in a training-free manner (for the attention mechanism itself) by leveraging physical motion priors and reusing pre-computed text embeddings, avoiding the need for invasive architectural changes. Deployment Optimization Strategies. Our work integrates several orthogonal efficiency techniques. Similar to TeaCache ( 20 ) and DeepCache ( 22 ) which skip layers to save compute, we reduce the intra-layer attention cost. To ensure temporal coherence under aggressive pruning, we employ Parameter-Efficient Fine-Tuning (PEFT) akin to LoRA ( 14 ) , but specifically target projection layers sensitive to sparsity. Finally, instead of costly online neural architecture search (NAS) ( 29 ) , we utilize an offline Bayesian Optimization pipeline to pre-compile a lightweight lookup table, enabling instantaneous adaptation at runtime. 3 Method Figure 1: Overview of DynamicRad. Offline BO builds a lookup table of motion-regime-specific sparsity configurations. At inference time, a semantic router selects a regime from the prompt embedding, which determines either static-ratio or dynamic-threshold selection within a shared structured candidate set to produce the final block-sparse mask. An optional mask-aware LoRA compensates for sparsity-induced information loss. Radial Attention reduces attention cost via a static energy-decay mask ( 19 ) , but its fixed sparsity pattern can be brittle under long horizons and complex motion where long-range dependencies become critical. To resolve these limitations, we propose DynamicRad , a unified sparse-attention paradigm summarized in Figure 1 . The framework combines offline BO-based configuration, prompt-conditioned regime routing, and a shared candidate set with two selection modes: static-ratio for throughput-first deployment and dynamic-threshold for quality-first deployment. An optional mask-aware LoRA further compensates for sparsity-induced degradation. Both modes share the same kernel-friendly block aggregation logic, and the following subsections present the formulation. 3.1 Dynamic Neighborhood Partitioning and Parameter Space To ground the decisions of our Semantic Router within a mathematically rigorous framework, we formalize a multi-dimensional parameter space that governs the attention connectivity. Rather than treating these as static hyperparameters, we