Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer

Mohsen Ghafoorian, Denis Korzhenkov, Amirhossein Habibian · 2025

Read Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer by Mohsen Ghafoorian, Denis Korzhenkov, Amirhossein Habibian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24899.txt.

Introduction

Mohsen Ghafoorian, Denis Korzhenkov, and Amirhossein Habibian present a method that swaps costly attention in video generation models for a cheaper form, in a paper from 2025. The catalog title appends a sponsor thanks note, which the head title omits. Standard attention cost grows fast with video length and resolution.

Their recipe mixes two attention styles token by token and adapts a trained model with light fine tuning over a few days of computing. A cost aware plan decides which layers keep the richer attention. On a leading video model, quality stayed competitive on standard benchmarks and human ratings. Phone tests showed better scaling for longer videos in speed, memory, and compute.

Catalog record

CREATORMohsen Ghafoorian, Denis Korzhenkov, Amirhossein Habibian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Mohsen Ghafoorian, Denis Korzhenkov, Amirhossein Habibian Abstract Transformer-based video diffusion models (VDMs) deliver state-of-the-art video generation quality but are constrained by the quadratic cost of self-attention, making long sequences and high resolutions computationally expensive. While linear attention offers sub-quadratic complexity, prior attempts fail to match the expressiveness of softmax attention without costly retraining. We introduce Attention Surgery , an efficient framework for linearizing or hybridizing attention in pretrained VDMs without training from scratch. Inspired by recent advances in language models, our method combines a novel hybrid attention mechanism—mixing softmax and linear tokens—with a lightweight distillation and fine-tuning pipeline requiring only a few GPU-days. Additionally, we incorporate a cost-aware block-rate strategy to balance expressiveness and efficiency across layers. Applied to Wan2.1 1.3B, a state-of-the-art DiT-based VDM, Attention Surgery achieves the first competitive sub-quadratic attention video diffusion models, reducing attention cost by up to 40% in terms of FLOPs, while maintaining generation quality as measured on the standard VBench and VBench-2.0 benchmarks. 1 Introduction Video diffusion models (VDMs) have become a cornerstone of generative modeling, enabling high-fidelity, temporally coherent video synthesis for applications from entertainment to simulation. Early VDMs relied on U-Net backbones, but these architectures struggle to scale and capture long-range temporal dependencies. Recent advances favor Diffusion Transformers (DiTs) ( Peebles & Xie 2023 ) . These models operate on spatiotemporal patches and provide global receptive fields from the outset. State-of-the-art systems such as Wan2.1 ( Wan et al. 2025 ) , CogVideoX ( Yang et al. 2025 ) , HunyuanVideo ( Lab 2025 ) , PyramidalFlow ( Jin et al. 2025 ) , and Open-Sora Plan ( Lin et al. 2024 ) exemplify this trend, consistently outperforming U-Net-based models in quality and scalability. Recent surveys confirm this transition, highlighting DiTs as the dominant architecture for video generation ( Wang et al. 2025b ; Melnik et al. 2024 ) . While recent DiT-based video diffusion models deliver state-of-the-art quality, they come with substantial computational and memory costs that limit their practical applicability. A primary bottleneck lies in the self-attention mechanism, whose complexity scales quadratically with the sequence length, i.e., O(N^{2}d) in time and O(N^{2}) in memory, where N denotes the number of tokens and d the hidden dimension ( Vaswani et al. 2017 ; Rabe & Staats 2021 ) . This issue is particularly severe in video diffusion, where the token count easily reaches tens of thousands due to the combination of spatial patches and multiple frames. Our profiling of large-scale DiT-based video diffusion models indicates that a vast proportion of compute is devoted to self-attention. For instance, in Wan2.1 1.3B, more than 76% of the total compute within the transformer blocks is attributable to self-attention alone. Even with optimizations such as FlashAttention ( Dao et al. 2022 ) , the quadratic scaling remains a fundamental barrier, constraining both training and inference when targeting higher resolutions, longer durations, or multi-shot videos. Consequently, reducing the attention cost without sacrificing model quality is critical for making video diffusion models more efficient and broadly deployable. Linear attn. no distill. Linear attn. with distill Hybrid attn. with distill. Wan 2.1 Original Figure 1: Impact of the proposed method components: attention distillation and hybrid attention. Prompt: ”An astronaut flying in space, Van Gogh style.” Although linear-time attention to address the quadratic cost of softmax attention ( Katharopoulos et al. 2020 ; Choromanski et al. 2021 ) has been around for several years, no competitive video diffusion model based on linear attention exists today. Three factors explain this gap. First , training such models from scratch is prohibitively expensive: state-of-the-art video diffusion systems require hundreds of thousands to millions of GPU hours and massive curated datasets ( Blattmann et al. 2023 ; Chen et al. 2024 ; Wan et al. 2025 ) . Second , there is no practical method proposed for distilling softmax attention into linear attention under reasonable compute budgets for video diffusion. This difficulty arises because the exponential kernel underlying softmax requires an infinite-dimensional feature map for exact representation, making efficient approximations challenging ( Han et al. 2024 ; Zhang et al. 2025 ) . While linear attention in image diffusion ( Li et al. 2023 ) and low-rank linearization in language models ( Zhang et al. 2025 ) show promise, no analogous solution exists for the more complicated spatiotemporal token interactions in video diffusion. Third , lower expressiveness in linear attention often results in notable degradation in the attention transformation fidelity and consequently lower quality generations, specifically for videos with more complicated temporal signal dynamics. To address these challenges, we propose an efficient attention surgery strategy – eliminating the need for extensive retraining from scratch – coupled with a novel efficient hybrid attention architecture inspired by recent developments in language modeling ( Zhang et al. 2025 ) . Intuitively, if a small subset of tokens retains full softmax attention while the rest use linear attention, the model can preserve global structure and fine-grained dependencies where needed, while scaling efficiently elsewhere. Our approach significantly narrows the quality gap between linearized and full softmax attention while achieving higher efficiency than the original softmax attention models. Importantly, it can be realized with modest compute –requiring less than 0.4k GPU hours for the overall surgery – making it practical for a wide range of research and industrial settings. We validate our method on Wan2.1 , a state-of-the-art video diffusion model, demonstrating that our contributions are successfully applicable to transformer-based diffusion models. Figure 1 illustrates the obtained advantage. Our main contributions are as follows: • We introduce attention surgery , an efficient recipe that enables achieving competitive linear/hybrid models within only a few GPU days training on modestly-sized training datasets, liberalizing the process of such significant architectural operations. • We propose a novel hybrid attention formulation with components carefully designed taking the intrinsics of videos into consideration. • We propose a novel block-rate optimization strategy that adjusts the attention configuration of each block based on its transformation complexity, achieving the best accuracy–efficiency trade-off within a given compute budget. 2 Related Work Efficient Attention. Numerous approaches have been proposed to reduce the quadratic complexity of self-attention, for perception e.g. EfficientViT ( Cai et al. 2023 ) , PADRe ( Letourneau et al. 2025 ) , Performer ( Choromanski et al. 2021 ) , and Linformer ( Wang et al. 2020 ) , for image generation e.g. SANA ( Li et al. 2023 ) , LinGen Wang et al. 2025a , Grafting ( Chandrasegaran et al. 2025 ) , and for language modeling ( Mercat et al. 2024 ; Wang et al. 2024 ; Yang et al. 2024 ; Zhang et al. 2024 ; Zhang et al. 2025 ) . Although these methods demonstrate the feasibility of sub-quadratic or more efficient attention, they typically require extensive training or training from scratch (e.g., SANA ( Li et al. 2023 ) ). In contrast, our work aims for lightweight distillation and fine-tuning of pre-trained softmax-attention-based models into an efficient hybrid attention design specifically tailored for video diffusion under modest compute budgets. Moreover, we target video generation task with unique challenges

Read the complete text