DiTVR: Zero-Shot Diffusion Transformer for Video Restoration

Sicheng Gao, Nancy Mehta, Zongwei Wu, Radu Timofte · 2025

Read DiTVR: Zero-Shot Diffusion Transformer for Video Restoration by Sicheng Gao, Nancy Mehta, Zongwei Wu, Radu Timofte (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07811.txt.

Introduction

Sicheng Gao, Nancy Mehta, Zongwei Wu, and Radu Timofte present a method to restore low-quality video without new training for each task, in a paper from 2025. Restoration covers sharpening, noise removal, and blur removal. Older methods often invent poor detail, while recent generative methods can flicker between frames.

Their system pairs a diffusion transformer with attention that follows motion paths across frames. A cache picks the most relevant tokens from nearby frames. The sampler enforces match with the input only in smooth image bands, which keeps fine detail and speeds convergence. It set new best scores for zero-shot restoration and stayed robust when motion was noisy.

Catalog record

CREATORSicheng Gao, Nancy Mehta, Zongwei Wu, Radu Timofte
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07811

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiTVR: Zero-Shot Diffusion Transformer for Video Restoration Sicheng Gao, Nancy Mehta, Zongwei Wu, Radu Timofte Abstract Video restoration aims to reconstruct high-quality video sequences from low-quality inputs, addressing tasks such as super-resolution, denoising, and deblurring. Traditional regression-based methods often produce unrealistic details and require extensive paired datasets, while recent generative diffusion models face challenges in ensuring temporal consistency. We introduce DiTVR , a zero‑shot video restoration framework that couples a diffusion transformer with trajectory‑aware attention and a wavelet‑guided, flow‑consistent sampler. Unlike prior 3D convolutional or frame‑wise diffusion approaches, our attention mechanism aligns tokens along optical‑flow trajectories, with particular emphasis on vital layers that exhibit the highest sensitivity to temporal dynamics. A spatiotemporal neighbour cache dynamically selects relevant tokens based on motion correspondences across frames. The flow-guided sampler injects data consistency only into low‑frequency bands, preserving high‑frequency priors while accelerating convergence. DiTVR establishes a new zero‑shot state of the art on video restoration benchmarks, demonstrating superior temporal consistency and detail preservation while remaining robust to flow noise and occlusions. Introduction Video restoration seeks to recover high-quality (HQ) sequences from degraded (LQ) inputs, spanning super-resolution ( Kappeler et al. 2016 ; Rota, Buzzelli, and van de Weijer 2023 ) , denoising ( Li et al. 2023 ; Ji, Lei, and Zhang 2023 ) , and deblurring ( Zhang et al. 2018 ; Zhou et al. 2019 ) . Current state-of-the-art methods fall into two families: regression-based approaches ( Caballero et al. 2017 ; Huang, Wang, and Wang 2017 ; Chan et al. 2021 ) and generative (diffusion-driven) approaches ( Lucas et al. 2019 ; Yi et al. 2020 ; Xu et al. 2024 ) . Regression-based methods, which often assume well-defined degradations, tend to produce outputs with unrealistic details. Consequently, they are often less effective for applications requiring high-fidelity restoration under varying and unpredictable degradation conditions. Additionally, these methods usually require large-scale paired datasets and involve training specific models for each degradation setting, making them impractical for real-world scenarios where degradation types are diverse and dynamically changing. Figure 1: Model inference process comparison with previous methods. (a) U-Net-based zero-shot methods ( Yeh et al. 2024 ) rely on local convolutions, aggregating motion only within a small window and therefore exhibiting flicker and fragmented dynamics, (b) Vanilla DiT with direct flow warping enlarges the receptive field but leaves tokens misaligned, introducing ghosting and other temporal artifacts, and (c) DiTVR overcomes both issues by pairing a flow-guided spatiotemporal neighbor cache with trajectory-aware attention injected solely into vital layers, restoring sharp details with superior temporal consistency. Diffusion generators ( Ramesh et al. 2022 ; Rombach et al. 2022 ; Ho et al. 2022 ) adapt flexibly to diverse degradations and can handle multiple restoration tasks with a single network ( Gao et al. 2023 ) . Yet naïvely extending image-based diffusers to video introduces flickering because each frame follows an independent noise path. Recent works retrofit latent diffusion models with temporal attention or 3-D convolutions ( Blattmann et al. 2023 ; Zhou et al. 2024 ; Yang et al. 2023c ; Rota, Buzzelli, and van de Weijer 2023 ; Chen et al. 2024 ) , but these additions are computationally heavy and rely solely on U-Net backbones that struggle with long-range dependencies. Methods such as DiffIR2VR-Zero ( Yeh et al. 2024 ) and ZVRD ( Cao et al. 2024 ) sidestep training entirely by injecting optical-flow guidance at inference. However, they employ U-Net architectures and use flow primarily for structural alignment, often neglecting fine-detail synthesis and global coherence. In this paper, we propose DiTVR, a novel zero-shot video restoration framework based upon Diffusion Transformers (DiTs) that circumvents both the limited temporal receptive fields of U-Net architectures (Fig. 1 (a)) and the spatial misalignment caused by naive optical flow warping in vanilla DiT (Fig. 1 (b)). Unlike prior U-Net-based methods, which struggle to enforce temporal consistency, our approach integrates explicit optical flow guidance with targeted enhancements to vital layers that are most sensitive to temporal dynamics, maintaining structural coherence across frames while preserving fine details. This ensures high-quality video reconstructions without the need for task-specific fine-tuning. Building on these insights, we develop DiTVR with three pivotal components. First, a Spatiotemporal Neighbor Selection Cache mechanism reduces memory overhead by storing and retrieving key-value representations from non-overlapping spatial blocks, dynamically selected based on optical flow trajectories to enhance long-range temporal modeling. This caching strategy enables efficient adaptation to arbitrary-resolution videos and also streamlines the subsequent temporal attention process by focusing on spatially and temporally relevant information. Second, a Trajectory-Aware Attention mechanism replaces simple frame-wise attention by aligning features along motion trajectories, with particular emphasis on vital layers that are most sensitive to temporal dynamics as identified through our layer-wise analysis. This targeted approach ensures stable and temporally coherent reconstructions while mitigating ghosting artifacts. Finally, a Flow-Guided Diffusion Sampling technique extends the reverse diffusion process by integrating optical flow guidance, allowing the model to refine motion details adaptively and enforce structural consistency across frames. To summarize, our main contributions are threefold: • A novel zero-shot diffusion transformer framework, DiTVR, guided by optical flow trajectories for coherent and high-fidelity video restoration. • We introduce a vital-layer analysis that identifies layers most sensitive to temporal dynamics and augments them with two flow-driven modules: Spatiotemporal Neighbor Cache and Trajectory-Aware Attention to improve temporal stability. Additionally, we propose a Flow-Guided Diffusion Sampler that operates during the sampling process to preserve fine-grained detail information and enforce motion consistency across frames. • Extensive experiments validate the effectiveness of our method. Our model achieves state-of-the-art performance on existing zero-shot video restoration benchmarks, especially in maintaining temporal consistency. Related Work Video Restoration Similar to image restoration ( Zhang et al. 2017 ) , video restoration aims to address issues such as noise, blur, and low resolution ( Chan et al. 2021 ; Chan et al. 2022 ; Li et al. 2023 ; Youk, Oh, and Kim 2024 ) . The existing video restoration methods need to be trained for every single task. However, acquiring sufficient paired training data for video restoration is significantly more difficult than for image restoration, and the reliance on predefined degradation processes ( Liang et al. 2022 ; Li et al. 2020 ; Liang et al. 2024 ) limits effectiveness in real-world scenarios with unknown and diverse degradations. Recent approaches such as Upscale-A-Video ( Zhou et al. 2024 ) , SATeCo ( Chen et al. 2024 ) , and MGLD-VSR ( Yang et al. 2023c ) extend pre-trained image diffusion models by incorporating and fine-tuning temporal layers. Unlike these trainable methods, we employ a pre-trained image diffusion model for video restoration with a training-free framework, enabling the generation of realistic details while performing the restoration. Diffusion Models for Image Restoration To improve the fidelity of details using pre-trained Stable Diffusion (SD) models, previous works ( Wang et al. 2024 ; Yang et al. 2023b ; Zhang and Agrawala 2023 ; Lin et al. 2023 ) have fine-tuned additional restoration modules. However, the latent space in these models is often susceptible to losing high-frequency image details ( Crowson et al. 2024 ) . Additionally, methods such as ( Fei et al. 2023 ; Wang et al. 2023 ; Kawar et al. 2022 ) have

Read the complete text