Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep

Tianyi Liu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau, [4pt] Nanyang Technological University Shanghai Jiao Tong University The Hong Kong Polytechnic University {liut0038, lu0001ye, e190210, gaoj0018}@[email protected]@ntu.edu.sg{yi-eie.wang, lap-pui.chau}@polyu.edu.hk · 2026

Read Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep by Tianyi Liu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau, [4pt] Nanyang Technological University Shanghai Jiao Tong University The Hong Kong Polytechnic University {liut0038, lu0001ye, e190210, gaoj0018}@[email protected]@ntu.edu.sg{yi-eie.wang, lap-pui.chau}@polyu.edu.hk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24260.txt.

Introduction

Liu and co-authors present HetCache, a training-free way to speed up diffusion-based video editing. Diffusion Transformers denoise step by step, which costs much compute. Past methods reuse features across time steps but ignore waste inside the model. Many attention operations over space-time tokens add little to the output. HetCache splits tokens into context and generative groups by spatial priors. It caches the context tokens that correlate most with the rest. Editing quality barely drops, while speed rises 2.67 times in their tests.

Catalog record

CREATORTianyi Liu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau, [4pt] Nanyang Technological University Shanghai Jiao Tong University The Hong Kong Polytechnic University {liut0038, lu0001ye, e190210, gaoj0018}@[email protected]@ntu.edu.sg{yi-eie.wang, lap-pui.chau}@polyu.edu.hk
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34639
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep Tianyi Liu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau, [4pt] Nanyang Technological University Shanghai Jiao Tong University The Hong Kong Polytechnic University {liut0038, lu0001ye, e190210, gaoj0018}@[email protected]@ntu.edu.sg{yi-eie.wang, lap-pui.chau}@polyu.edu.hk Abstract Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model’s output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67 × \times latency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality. 1 Introduction Figure 1 : (a). Illustration of the acceleration dimensions in Diffusion Transformers (DiTs). Unlike existing methods, the proposed Heterogeneous Caching (HetCache) jointly models denoising-step redundancy in the diffusion process and token redundancy within the Transformer backbone. (b). As a tailored heterogeneous strategy, HetCache accelerates diffusion-based masked video-to-video (MV2V) editing while maintaining generation quality. Diffusion-based generative methods have recently gained attention in various video editing tasks 29 . With Diffusion Transformers (DiTs), which adopt the Transformer as the denoising backbone, both visual quality and generalization ability have been significantly improved in video synthesis and editing 7 ; 21 . Through scalable parameterization, DiTs provide larger modeling capacity and finer spatio-temporal representations, enabling more flexible generation across complex scenes 25 . However, these advantages come with substantial computational cost. The dense interactions among spatio-temporal tokens within the Transformer layers lead to high computational complexity, while the iterative nature of the denoising process in diffusion models requires repeated network forward evaluation over multiple timesteps, resulting in significant inference latency. These factors collectively constrain the real-time and interactive applications for diffusion-based video editing. As the demand for efficient and lightweight video generation continues to grow, recent studies on accelerating video diffusion models have explored knowledge distillation 22 and post-training optimization methods 24 . These approaches typically rely on knowledge transfer between teacher and student models or weight quantization to reduce inference cost, but they inevitably require additional training and data resources, resulting in increased computational overhead. To eliminate retraining costs, subsequent work has investigated training-free acceleration strategies, among which feature caching has gained particular attention. By caching and reusing intermediate features in denoising timesteps, such methods achieve acceleration within the diffusion framework without modifying model parameters 36 ; 18 ; 35 . However, existing approaches primarily focus on temporal redundancy across timesteps, while neglecting the token redundancy and heterogeneity introduced by the viideo editing task and additional temporal dimension in Transformer-based video models. This omission limits both the flexibility and the upper bound of current caching-based acceleration schemes. In this work, we aim to develop a more efficient feature caching mechanism for video diffusion models tailored to masked video-to-video (MV2V) generation and editing tasks. Our design considers two complementary sources of redundancy including 1) timestep-level redundancy in the denoising process, and 2) spatio-temporal token redundancy within the attention layers of Diffusion Transformers (DiTs). The main challenge lies in identifying redundant tokens in a video editing context where spatial and temporal dependencies are highly uneven. Unlike general video generation, the MV2V task setting features explicit regions of interest (ROI) 11 . Therefore, applying uniform caching to all tokens within a denoising step for computation can degrade the reconstruction quality inside the masked area. Intuitively, the attention mechanism should allow a minimal number of context(unmasked) tokens to provide strong semantic guidance for the generative(masked) tokens, ensuring sufficient representation quality while reducing computational cost. However, the representational importance and interaction strength of context tokens are only observable after attention computation. Without an effective mechanism to estimate these properties beforehand, token sampling may negatively affect the quality of generated content. To address this problem, we propose Heterogeneous Caching (HetCache), a training-free caching strategy designed for efficient inference of video editing. The key idea of HetCache is that both denoising timesteps and context tokens in Diffusion Transformers (DiTs) contribute unequally to the final generation quality. By modeling this heterogeneity across temporal and token dimensions, HetCache performs selective caching that adapts to each dimension independently. During inference, HetCache first identifies anchor timesteps where model output is expected to change significantly and performs full computation at these steps. Within each anchor timestep, unmasked tokens are divided into two groups based on spatial priors: context tokens, which are subject to selection, and margin tokens, which are fully preserved around the masked boundary. These tokens are further clustered in the semantic space, and the attention interactions between context tokens and masked generative tokens are then analyzed to estimate their semantic relevance, allowing the model to identify informative context tokens. In subsequent timesteps, the cached representative tokens replace the full set of context tokens during the attention computation, forming partial computing steps. This design effectively reduces the number of active tokens without compromising generation fidelity, thus achieving acceleration for diffusion-based video editing. The contributions of this work are summarized below. • Token analysis for diffusion-based video editing. We analyze the token-wise redundancy in DiT-based MV2V generation and editing, revealing the inherent token heterogeneity caused by the region-of-interest (ROI) nature. • A token-level caching mechanism for efficient diffusion-based video editing. We propose HetCache, a training-free caching framework that performs heterogeneous caching across both denoising timesteps and spatio-temporal tokens. By adapting caching and reuse strategies to the characteristics of each dimension, HetCache introduces partial denoising steps guided by expected output variation and reduces the attention computation through semantic representativeness and interaction-based selection. • Comprehensive evaluation and state-of-the-art efficiency. Extensive experiments and evaluation using common DiT backbones for video completion and text-guided MV2V editing on VACE-Benchmark and VPBench demonstrate that HetCache achieves an improved balance between generation quality and computational efficiency, providing a practical solution toward real-time and interactive diffusion-based video editing. 2 Related Works 2.1 Diffusion-based Video Editing Diffusion models have evolved from U-Net backbones 6 ; 23 to Diffusion Transformers (DiTs) 25 ; 2 ; 13 , improving scalability and generation quality, but also increasing inference cost. In practice, representative DiT systems report consistent quality gains at the cost of higher per-step compute, which amplifies the latency bottleneck under many sampling steps. In recent years, diffusion-based video editing can be viewed as a conditional video-to-video (V2V) generation problem 16 (often “MV2V”) with explicit guidance such as

Read the complete text