DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
Haowei Zhu, Ji Liu, Ziqiong Liu, Dong Li, Junhai Yong, Bin Wang, Emad Barsoum · 2026
Read DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity by Haowei Zhu, Ji Liu, Ziqiong Liu, Dong Li, Junhai Yong, Bin Wang, Emad Barsoum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03674.txt.
Introduction
Haowei Zhu and colleagues sped up diffusion transformers, the image generators that refine pictures over many steps, by learning which tokens each layer can skip. A learnable network plus a dynamic-programming solver sets sparsity per layer, and a two-stage training plan removes the need to run every step in full. On PixArt with 20 sampling steps, computing cost fell 54 percent while quality held or improved.
Catalog record
| CREATOR | Haowei Zhu, Ji Liu, Ziqiong Liu, Dong Li, Junhai Yong, Bin Wang, Emad Barsoum |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54119 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03674
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity Haowei Zhu, Ji Liu, Ziqiong Liu, Dong Li, Junhai Yong, Bin Wang, Emad Barsoum Abstract Diffusion models demonstrate outstanding performance in image generation, but their multi-step inference mechanism requires immense computational cost. Previous works accelerate inference by leveraging layer or token cache techniques to reduce computational cost. However, these methods fail to achieve superior acceleration performance in few-step diffusion transformer models due to inefficient feature caching strategies, manually designed sparsity allocation, and the practice of retaining complete forward computations in several steps in these token cache methods. To tackle these challenges, we propose a differentiable layer-wise sparsity optimization framework for diffusion transformer models, leveraging token caching to reduce token computation costs and enhance acceleration. Our method optimizes layer-wise sparsity allocation in an end-to-end manner through a learnable network combined with a dynamic programming solver. Additionally, our proposed two-stage training strategy eliminates the need for full-step processing in existing methods, further improving efficiency. We conducted extensive experiments on a range of diffusion-transformer models, including DiT-XL/2, PixArt- α \alpha , FLUX, and Wan2.1. Across these architectures, our method consistently improves efficiency without degrading sample quality. For example, on PixArt- α \alpha with 20 sampling steps, we reduce computational cost by 54 % 54\% while achieving generation metrics that surpass those of the original model, substantially outperforming prior approaches. These results demonstrate that our method delivers large efficiency gains while often improving generation quality. 1 Introduction In recent years, diffusion models have made remarkable progress in the field of image generation. Among them, the Stable Diffusion series ( 36 ; 34 ; 44 ; 10 ) has achieved significant success in high-quality image generation ( 63 ; 61 ) . This advancement is largely attributed to the effectiveness of diffusion probabilistic models (DPM) ( 16 ) and the powerful U-Net ( 37 ) architecture, which allows high resolution synthesis with detail preservation. Additionally, some recent works ( 33 ; 5 ; 44 ) have explored the integration of diffusion models with Transformer-based architectures, demonstrating outstanding performance. In particular, scaling laws have been leveraged to expand the model size of Transformers ( 45 ) , further enhancing precision and generative quality, these large-scale models leverage improved expressivity and enhanced generalization, pushing the boundaries of generative artificial intelligence. However, despite these advances, the substantial computational cost associated with diffusion models presents a significant challenge for real-world deployment. The inference of such large models requires extensive computational resources, which can hinder practical applications. Addressing this issue requires innovations in model acceleration techniques to enable broader accessibility and usability of diffusion-based generative models. Existing methods of diffusion model acceleration typically focus on sampler optimization ( 41 ; 28 ) , model pruning ( 12 ; 56 ; 11 ) , distillation ( 52 ; 29 ; 39 ) , and feature caching ( 40 ; 31 ; 22 ; 23 ) . Feature caching methods leverages temporal redundancy to reuse intermediate features, achieving significant speedups. They become popular in the field of diffusion model acceleration due to non-training diffusion model and easy integrating into the original inference pipeline. Previous methods cache and reuse coarse-grained, layer-level features, whereas token cache methods ( 64 ; 65 ; 57 ) reuse token-level features, achieving better acceleration performance. However, these approaches require manual sparsity allocations and hand-crafted schedules that preserve several full forward passes during denoising, which limits the acceleration potential of token-level feature caching. To address these challenges, we propose DiffSparse, a learnable framework for optimizing layer-wise sparsity allocation in diffusion transformer models. Our approach dynamically determines the optimal sparsity configuration across all layers and inference steps, ensuring that the overall pruning rate is met in an end-to-end manner through a model-driven process. Moreover, DiffSparse eliminates the need for complete forward computations in predefined steps required by existing methods, further enhancing efficiency. Specially, our approach formulates the token cache optimization as a dynamic programming-based sparsity allocation problem. We innovatively design a learnable sparsity cost predictor, which predicts a cost matrix that quantifies the sparsity costs associated with target sparsity rates for all layers across every denoising step. Then we propose a dynamic programming approach to determine the optimal sparsity configurations for all layers over the relevant denoising steps, minimizing the overall sparsity cost while satisfying the required sparsity rate. Finally, we introduce a token selector that dynamically selects a specific proportion of tokens for reuse, leveraging the learned sparsity ratio to accelerate inference. To optimize the learnable sparsity cost predictor, we utilize a perceptual distillation loss that minimizes the degradation in generation quality. Furthermore, we introduce a two-stage training strategy that eliminates the need for complete forward computations in predefined steps required by existing methods while also improving accuracy. We have conducted extensive experiments on various transformer-based baselines, and the pruning results outperform other SOTA pruning methods by a large margin. For example, pruning 54\% of tokens yields an FID of 27.79, our method substantially better than the state-of-the-art methods ToCa (28.35) and TaylorSeer (29.08), while achieving a higher speedup (1.91 \times ) on PixArt- \alpha . These results underscore the practical effectiveness of our method. Our contributions are summarized as follows: • We propose DiffSparse, a differentiable approach to optimize layer-wise token sparsity in diffusion models sampling process. By integrating a sparsity cost predictor, dynamic programming solver, and adaptive token selector, it automates sparsity allocation and token reuse without manual heuristics. • We introduce a two-stage training strategy that eliminates the need for predefined complete forward computations in several steps required by existing methods, fully unlocked the acceleration potential of token-level feature caching. • Extensive experiments on diverse foundation models prove that our method surpasses existing SOTA methods by a large margin, setting new efficiency-accuracy benchmarks. 2 Related Work Diffusion Transformer Models. The integration of transformers into diffusion models has significantly advanced generative modeling, improving scalability and performance. Diffusion models, which generate data by iteratively denoising from a noise distribution. Traditionally, diffusion models relied on CNNs, but recent studies demonstrate the effectiveness of transformers ( 33 ; 5 ; 44 ; 2 ; 4 ; 49 ) . Diffusion Transformer (DiT) ( 33 ) replaces the U-Net backbone with a transformer, leveraging long-range dependencies and efficient scaling to achieve superior image generation. PixArt ( 5 ) builds on this by introducing a hierarchical transformer architecture and a novel noise schedule, excelling in high-resolution and text-to-image synthesis. Although diffusion transformer models have achieved great success, the substantial computational overhead from the iterative denoising process makes them inefficient for industrial deployment. Acceleration of Diffusion Models. Diffusion acceleration is a critical research area focused on reducing computational costs and improving inference efficiency while preserving high-quality generation. Recent advancements can be categorized into sampler optimization ( 42 ; 28 ; 27 ) , model pruning ( 12 ; 56 ) , distillation ( 39 ; 51 ) , and feature caching ( 20 ; 31 ; 62 ) . Sampler optimization reduces the number of denoising steps during inference using deterministic or adaptive strategies to approximate the denoising process efficiently. Model pruning removes redundant parameters and achieving speedups with structured pruning ( 11 ) . Other strategies, such as Rectified Flow ( 25 ) and