VipDiff : Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models

Chaohao Xie, Kai Han, Kwan-Yee K. Wong · 2025

Read VipDiff : Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models by Chaohao Xie, Kai Han, Kwan-Yee K. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12267.txt.

Introduction

Chaohao Xie, Kai Han, and co-authors fill missing regions in videos with a diffusion model that needs no extra training, in work from twenty twenty-five. Flow guided inpainting copies pixels from other frames, but big holes have no matching pixels at their center. Diffusion models invent rich content, yet naive use breaks time consistency.

Their framework steers the denoising process with valid pixels pulled along optical flow paths. Those pixels constrain the random noise the model starts from. Filled frames then feed further propagation and generation rounds. Different noise seeds give varied but coherent results. Tests beat leading video inpainting methods on both time coherence and fidelity, the authors report.

Catalog record

CREATORChaohao Xie, Kai Han, Kwan-Yee K. Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VipDiff : Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models Chaohao Xie, Kai Han, Kwan-Yee K. Wong Abstract Recent video inpainting methods have achieved encouraging improvements by leveraging optical flow to guide pixel propagation from reference frames, either in the image space or feature space. However, they would produce severe artifacts when the masked area is too large and no pixel correspondences could be found. Recently, denoising diffusion models have demonstrated impressive performance in generating diverse and high-quality images, and have been exploited in a number of works for image inpainting. These methods, however, cannot be applied directly to videos to produce temporal-coherent inpainting results. In this paper, we propose a training-free framework, named VipDiff , for conditioning diffusion model on the reverse diffusion process to produce temporal-coherent inpainting results without requiring any training data or fine-tuning the pre-trained models. VipDiff takes optical flow as guidance to extract valid pixels from reference frames to serve as constraints in optimizing the randomly sampled Gaussian noise, and uses the generated results for further pixel propagation and conditional generation. VipDiff also allows for generating diverse video inpainting results over different sampled noise. Experiments demonstrate that our VipDiff outperforms state-of-the-art methods in terms of both spatial-temporal coherence and fidelity. 1 Introduction Video inpainting aims to generate spatial-temporal coherent contents for the masked areas in corrupted video frames. Existing video inpainting methods can be broadly classified into 1) end-to-end synthesis methods and 2) flow-based pixel propagation methods. End-to-end synthesis methods typically adopt 3D convolutions [ 4 , 3 , 12 ] , attention modules [ 17 , 18 , 41 ] , or multi-frame transformer-based networks [ 41 , 20 , 19 , 47 ] . They take corrupted frames and their corresponding masks as input, and train a video completion network to directly output the completed frames. Since they have fixed temporal receptive windows (i.e., only a fixed length of reference frames can be fed to the input), severe artifacts and temporal incoherence often occur when the input frames fail to provide useful texture and content hints for the frame completion models (see Fig. 1 (b)). Flow-based pixel propagation methods first conduct optical flow completion and then utilize the completed flows to guide a pixel propagation step, which warps valid pixels from reference frames to a target frame. An additional synthesis network is trained to fill the remaining masked pixels. Although these flow-based methods [ 39 , 8 , 43 , 14 ] can achieve better temporal coherence over the generated frames, their results still show obvious artifacts in the mask center when the masks are large (see Fig. 1 (c)). Undoubtedly, there is still a large room for improvement. Recently, denoising diffusion models [ 28 , 11 ] have attracted enormous attention due to their impressive performance in generating diverse and high-quality images from Gaussian noise through a series of denoising steps. There are several methods, like DPS [ 5 ] and ReSampling [ 31 ] , which adopt pre-trained diffusion model for image inpainting via sampling posterior distributions. RePaint [ 21 ] introduces a time travel strategy to improve spatial coherence between masked and unmasked areas, and CoPaint [ 42 ] proposes optimizing the randomly sampled Gaussian noise using the unmasked pixels as constraints. Latent Diffusion Models (LDM) [ 27 ] map the input to a low-dimensional space with an encoder and feed the encoded features for conditional generation in the latent feature space. These diffusion-based methods work well for the image inpainting task. However, they cannot be applied directly to videos since they do not consider temporal coherence. For instance, we show the inpainting results of LDM on three frames randomly selected from two videos in Fig. 1 (d). Even though the surrounding patterns are similar, LDM generates different contents in the masked area in different frames and fails to maintain temporal coherence. (a) Input (b) ProPainter [ 47 ] (c) ECFVI [ 14 ] (d) LDM [ 27 ] (e) Ours-Sample1 (f) Ours-Sample2 (g) Ours-Sample3 Figure 1: Video inpainting results on Davis dataset, we randomly select 3 frames from ’bear’ and ’cows’ videos. LDM generates different contents, ProPainter and ECFVI contain artifacts in the mask center, while our samples are more sharp and temporal-coherent. Inspired by the above diffusion-based image inpainting methods, we propose a training-free framework for conditioning pre-trained image-level diffusion models to generate spatial-temporal coherent and diverse video inpainting results. Given a corrupted video, we first adopt a flow completion model to predict all the optical flows between frames. We then use the completed flows to propagate valid pixels from reference frames to a target frame. Next, we carry out the reverse diffusion process, where valid pixels (i.e., both pixels in the unmasked area and pixels propagated from other frames) in the target frame serve as constraints in optimizing the sampled noise through backpropagation. After optimization, a spatial-temporal coherent frame is generated. Its pixels can be propagated to other frames again and the process is repeated for the next target frame. Note that once we generate one completed frame and propagate its pixels to other frames, the number of remaining invalid (masked) pixels in most frames, especially those nearby ones, would become zero. Since we only need to perform conditional generation on those frames with a non-zero size mask, our framework is therefore very time efficient. VipDiff can generate both spatial-temporal coherent results and diverse inpainted videos. In Fig. 1 (e), (f) and (g), we show three different results on two videos. All of our generated results are coherent with the video motions. VipDiff has the following advantages. 1) It eliminates the efforts in collecting large-scale video datasets and training a large video diffusion model, which may only be possible to be conducted by large companies or foundations. 2) It leverages the diverse generation capability of pre-trained image-level diffusion models and generates results far better than SOTA methods, while still allows users to choose from diverse video inpainting results. 3) It just takes several minutes to inpaint a video sequence of about 100 to 200 frames on a single RTX 3090 GPU, demonstrating that it is both device friendly and time efficient. Hence our method enables users with limited GPU resources to utilize the ability of pre-trained diffusion models for video inpainting. We summarize our contributions as follows: • A training-free framework, named VipDiff , for conditioning pre-trained image-level diffusion model to generate spatial-temporal coherent video inpainting results. On large mask cases, it is capable of generating diverse results for users. • VipDiff saves the extensive computation and time costs in specifically training a video denoising diffusion model for video inpainting task. • To the best of our knowledge, we present the first work that successfully tames pre-trained image-level diffusion models for video inpainting task. • Experiments on public datasets demonstrate superior performance of VipDiff over existing state-of-the-arts. 2 Related Works In this section, we give a brief review of video inpainting methods, including recent methods based on diffusion probabilistic model [ 28 ] . 2.1 Patch-based Methods Traditional inpainting methods [ 1 ] analyze local structures and fluid dynamics [ 2 ] along the mask boundary, and exploit spatial-temporal similarities [ 24 , 36

Read the complete text