CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, Fatih Porikli · 2026
Read CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers by Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, Fatih Porikli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14191.txt.
Introduction
Li and four co-authors offer CoReDiT, a structured token pruning framework for diffusion transformers. They generate striking images and video but cost heavily, capping scale and on-device use. CoReDiT scores local redundancy in the latent token lattice with a linear-time spatial coherence score, skipping extra tokens in self-attention. Skipped attention outputs are rebuilt by pooling spatially neighboring retained tokens, preserving a dense representation. A progressive, block-adaptive schedule ramps pruning up, giving bigger budgets to blocks and denoising steps at higher overlap.
Catalog record
| CREATOR | Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, Fatih Porikli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34566 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14191
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, Fatih Porikli Abstract Diffusion Transformers (DiTs) deliver remarkable image and video generation quality but incur high computational cost, limiting scalability and on-device deployment. We introduce CoReDiT, a structured token pruning framework for DiTs across vision tasks. CoReDiT uses a linear-time spatial coherence score to estimate local redundancy in the latent token lattice and skips high-coherence (redundant) tokens in self-attention. To maintain a dense representation and avoid visual discontinuities, we reconstruct skipped attention outputs via coherence-guided aggregation of spatially neighboring retained tokens. We further introduce a progressive, block-adaptive pruning schedule that increases pruning gradually and allocates larger budgets to blocks and denoising steps with higher redundancy. Across state-of-the-art diffusion backbones including PixArt- α \alpha and MagicDrive-V2, CoReDiT achieves up to 55% self-attention FLOPs reduction and inference speedups of 1.33 × \times on cloud GPUs and 1.72 × \times on mobile NPUs, while maintaining high visual quality. Notably, CoReDiT also increases on-device memory headroom, enabling higher-resolution generation. 1 Introduction Diffusion models learn to synthesize data through an iterative denoising process, and have achieved state-of-the-art results across various generative applications, such as conditional image generation, image editing (inpainting and super-resolution) and video synthesis. Building on this success, Diffusion Transformers (DiTs) 21 replace convolutional U-Nets with attention-based backbones that effectively model long-range dependencies among tokenized patches; this architecture scale effectively with model size and facilitate the incorporation of conditioning from various modalities (e.g., class labels, text, and depth maps). However, these benefits come with demanding computational cost. Self-attention scales quadratically with the number of tokens, making high-resolution images and multi-frame videos particularly expensive and often impractical on compute- and memory-constrained mobile platforms. Empirically, a large fraction of tokens correspond to visually redundant or low-saliency regions (e.g., uniform backgrounds and smooth textures), suggesting substantial room for compute reduction. Prior work reduces inference cost by pruning or merging tokens using attention-based saliency 12 ; 27 , feature similarity 1 , or learned routing/predictors 23 ; 34 . However, several challenges limit their applicability to diffusion inference: (1) localizing low-saliency tokens efficiently and effectively, (2) preserving visual semantics for the tokens that do not participate in attention, and (3) determining a pruning schedule that adapts across transformer blocks and denoising timesteps. Contributions: To address these challenges, we propose CoReDiT, a token pruning framework for accelerating pre-trained DiTs that targets self-attention while preserving output fidelity. CoReDiT consists of three components: (1) Spatial coherence-based selector ( Section 3.1 ): We observe that redundant tokens in low-saliency regions are highly similar to their spatial neighbors. To exploit this, we partition the token lattice into small, non-overlapping grids and estimate a spatial coherence score for each token, based on its feature similarity to tokens in the same grid. This score can be implemented with linear-time, hardware-friendly reductions (e.g., grid means and dot products), adding minimal overhead while reliably selecting redundant tokens (with high coherence) to bypass attention. (2) Coherence-based reconstruction ( Section 3.2 ): Skipping tokens can break local spatial continuity, leading to artifacts in the generated outputs. To preserve a dense token lattice for subsequent blocks, we reconstruct skipped tokens from retained neighbors using similarity-weighted aggregation within a local neighborhood. This content-aware interpolation (unlike zeroing or naive forwarding) preserves visual semantics and mitigates artifacts while maintaining compatibility with the original DiT architecture. (3) Progressive, block-adaptive pruning ( Section 3.3 ): Pruning tolerance varies across blocks and timesteps: different blocks carry varying semantic importance, and late diffusion steps are typically less tolerant than early ones 34 . Hence, we fine-tune with a progressive schedule that automatically allocates the pruning budget to the most redundant blocks. At regular intervals during fine-tuning, we estimate a timestep-weighted redundancy score for each transformer block and increment the pruning ratio of the block with the highest redundancy; gradually increasing the pruning ratio enables fast recovery after each update and ensures stable adaptation. Combining these components results in an efficient pruning pipeline that reduces attention computation while preserving high-fidelity generation. Across text-to-image and video generation, CoReDiT enables higher-resolution synthesis and stable conditional alignment, achieving up to 55% reduction in self-attention FLOPs and wall-clock speedups of 1.33 \times on cloud GPUs and 1.72 \times on mobile devices. 2 Related Work Diffusion Models. Diffusion models have demonstrated impressive generative performance across image, video, and 3D domains. Early works commonly adopt U-Net backbones 25 ; 11 , and latent diffusion 24 improves scalability by denoising in a compressed latent space. Recently, Diffusion Transformers (DiTs) 21 replace U-Net architectures with stacks of transformer blocks, offering better scalability to high resolution and flexible conditioning, but incurring high inference cost from quadratic attention and multi-step denoising. Efficient Diffusion Transformers. Prior studies have improved the efficiency of DiTs along several complementary directions: (1) Token reduction within a timestep . Training-free token reduction often merges or downsamples tokens to reduce the effective sequence length, including token merging methods (e.g., ToMe 2 ) and diffusion-specific variants (e.g., ToMA 18 ) that aim to reduce merge overhead. Other training-free strategies reduce attention cost by downsampling only key/value tokens (e.g., ToDo 28 ). Learning-based methods learn per-layer/timestep keep ratios or token routers for content generation, including DiffCR 34 and dynamic token-density schemes such as FlexDiT 5 . A key challenge in DiTs (vs. ViTs) is that naively discarding tokens can introduce spatial discontinuities (texture breaks, boundary artifacts), motivating approaches that either preserve structure via merge/unmerge or explicitly reconstruct skipped content. (2) Cross-timestep caching and feature reuse . Caching-based methods exploit temporal redundancy across denoising steps by reusing intermediate features with selective refresh 31 ; 19 . Recent DiT-specific works further cache at finer granularity (e.g., token-wise feature caching in ToCa 39 ). These approaches primarily target cross-step redundancy and are conceptually orthogonal to within-step token reduction. (3) Dynamic computation and conditional routing. Dynamic DiT architectures adjust compute across timesteps and/or spatial regions, e.g., DyDiT 38 uses timestep-wise dynamic width and spatial token skipping, while DiffCR 34 fine-tunes token routers with differentiable compression ratios. Compared to token reduction, these methods introduce explicit routing modules and typically optimize a dynamic compute graph. (4) Sparse/structured attention kernels. Another line accelerates DiTs by replacing full attention with sparse attention patterns and hardware-efficient kernels, including training-free profiling-based sparse attention (e.g., Sparse VideoGen 32 ) and trainable sparse attention (e.g., VSA 35 ). These methods change the attention operator itself rather than reducing the token set, and can be complementary to token pruning/merging. (5) Weight-pruning and architecture editing . Beyond tokens, structured and unstructured pruning remove channels, heads, or even full DiT blocks, with brief recovery fine-tuning or lightweight calibration to retain quality 8 . Other work (e.g., grafting 4 ) edits the architectures of pretrained DiTs to explore more efficient backbones under small compute budgets; these approaches are naturally complementary to token sparsity methods. Positioning of CoReDiT. CoReDiT is a post-training method that targets within-timestep redundancy in DiT latent grids: it performs structured token skipping guided by local spatial coherence, and reconstructs skipped tokens to preserve a full lattice for subsequent layers. This differs from caching-based methods (e.g., FasterCache/ToCa) that exploit cross-timestep reuse