Time-adaptive Video Frame Interpolation based on Residual Diffusion
Victor Fonte Chavez, Claudia Esteves, Jean-Bernard Hayet · 2025
Read Time-adaptive Video Frame Interpolation based on Residual Diffusion by Victor Fonte Chavez, Claudia Esteves, Jean-Bernard Hayet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05402.txt.
Introduction
Fonte Chavez, Esteves, and Hayet filled missing video frames in this paper from twenty twenty-five. Frame interpolation invents the in-between images that make motion smooth. Hand-drawn animation is the hard case, since timing swings far more than in live video. Their method builds on diffusion models that refine an image step by step.
The model takes the target moment as an explicit input and re-estimates it during training. A scheme borrowed from super-resolution work keeps the step count near ten. The random nature of diffusion also yields a per-pixel doubt map for each frame. Tests on animation beat leading models. The authors release the code openly.
Catalog record
| CREATOR | Victor Fonte Chavez, Claudia Esteves, Jean-Bernard Hayet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Time-adaptive Video Frame Interpolation based on Residual Diffusion Victor Fonte Chavez, Claudia Esteves, Jean-Bernard Hayet Abstract In this work, we propose a new diffusion-based method for video frame interpolation (VFI), in the context of traditional hand-made animation. We introduce three main contributions: The first is that we explicitly handle the interpolation time in our model, which we also re-estimate during the training process, to cope with the particularly large variations observed in the animation domain, compared to natural videos; The second is that we adapt and generalize a diffusion scheme called ResShift recently proposed in the super-resolution community to VFI, which allows us to perform a very low number of diffusion steps (in the order of 10 10 ) to produce our estimates; The third is that we leverage the stochastic nature of the diffusion process to provide a pixel-wise estimate of the uncertainty on the interpolated frame, which could be useful to anticipate where the model may be wrong. We provide extensive comparisons with respect to state-of-the-art models and show that our model outperforms these models on animation videos. Our code is available at https://github.com/VicFonch/Multi-Input-Resshift-Diffusion-VFI A Preprint Keywords Video frame interpolation, Diffusion models, Deep learning \mathbf{I}_{0}+\mathbf{I}_{1} GT Eisei SoftSplat Ours Figure 1: Zoomed-in qualitative comparison on a challenging interpolation scenario. From left to right: initial frame \mathbf{I}_{0} , final frame \mathbf{I}_{1} , ground truth (cropped), SoftSplat prediction, and our result. Note the quality of the reconstructed fingers for the different methods. 1 Introduction Traditional 2D animation requires a significant manual effort, as animators must create approximately twelve individual illustrations for every second of movement. This workflow demands large production teams, each with specialized training to generate and colorize a massive number of frames needed to produce an animated short film. With the growing global demand for traditional animation, studios face increasing pressure to produce high-quality content efficiently. In this context, Video Frame Interpolation (VFI) emerges as a key possible tool, with applications ranging from motion smoothing to frame rate enhancement. Traditional VFI methods based on optical flow and motion estimation struggle with occlusions, lighting variations, and complex motion, leading to artifacts such as blurring and ghosting. Recent advances in deep learning, particularly diffusion models, offer a promising alternative by effectively approximating high-dimensional data distributions. This work proposes a computationally efficient diffusion-based VFI model that integrates with traditional methods. Additionally, we propose to estimate and encode systematically the temporal position of the intermediate frame, mitigating training biases and inaccuracies and enabling flexible interpolation at arbitrary positions. Our approach aims to establish a scalable and adaptable diffusion framework for frame interpolation, addressing key limitations of existing techniques and paving the way for real-time applications. 2 Video Frame Interpolation: Overview of previous work For a few years now, deep learning-based methods have been forming the bulk of the state of the art in VFI, with most works using convolutional, encoder-decoder architectures to perform interpolation. We describe some of them hereafter. In [ 13 ] , to synthesize the video frames, Deep Voxel Flow (DVF) interpolates the pixel values from the frames that are close to the one to interpolate, by applying a voxel flow layer across space and time in the input video. Trilinear interpolation across the input video volume generates the final pixel value. The deep bidirectional predictive network (BiPN) [ 3 ] is a convolutional encoder-decoder network trained to regress the missing intermediate frames from two opposite directions, with a bi-directional encoder-decoder that simultaneously predicts the future-forward from the starting frame and predicts the past-backward from the ending frame. Multiple missing frames can be predicted by the decoder after taking the feature representations as input. PhaseNet [ 15 ] estimates the phase decomposition of the intermediate frame. It is designed as a decoder-only network, increasing its resolution level by level. The input is the response from the steerable pyramid decomposition of the two input frames, consisting of the phase and amplitude values for each pixel at each level. Each resolution level has a PhaseNet block that takes the decomposition values from the input images as its input, altogether with the resized feature maps and the resized predicted values from the previous level and it outputs the decomposition values of the intermediate image, from which the intermediate image is reconstructed. In [ 11 ] , the authors propose a solution for variable-length multi-frame video interpolation, with motion interception and occlusion jointly modeled. Bidirectional optical flows between input images are computed using a U-Net and are linearly combined to approximate the intermediate optical flows. The approximated optical flows are refined using another U-Net that also predicts soft visibility maps. The two input images are warped and linearly joined to form intermediate frames. In [ 1 ] , the Depth-Aware video frame Interpolation (DAIN) model detects occlusion through depth information. A depth-aware flow projection layer synthesizes flows that sample closer objects more often than those far away. The output frame is generated by warping the input frames, depth maps, and contextual features based on the optical flow and the local interpolation kernels. In [ 19 , 18 ] , the authors introduce a spatially-adaptive separable convolution technique to interpolate the intermediate frames. Pixelwise, it estimates a pair of 2D convolution kernels (four 1D kernels) to convolve the two video frames and compute the color of the output pixel. The pixel-dependent kernels capture both motion and re-sampling information required for interpolation. This idea is known as soft-splatting and has been at the core of many VFI methods. In [ 4 ] , the authors improve the soft-splatting idea with Adaptive Deformable Separable Convolution to adaptively estimate kernels, offsets and masks so that the network obtains information with fewer but more relevant pixels than in [ 19 ] . The learnable offsets make that pixels outside the local neighborhood can be reached, allowing to better handle large motion with smaller convolution kernels. As in many areas, generative modeling has been used in VFI, which can be cast as a conditional image generation problem. FIGAN [ 30 ] is a multi-scale generative adversarial network for frame interpolation, where the predicted flow and synthesized frame are constructed in a coarse-to-fine fashion. The network is jointly supervised at different levels with an adversarial and two content losses. A refinement module jointly processes the synthesized image with the original input frames that produced it. LDMVFI [ 6 ] is a latent diffusion model adapted from the latent diffusion models [ 23 ] . It includes an autoencoding model that projects images into a latent space, and a denoising U-Net that performs reverse diffusion in that latent space. The encoder to the latent space is a VQ-VAE that differs from the original VQ-GAN [ 8 ] in its use of MaxViT-based cross attention [ 29 ] for the feature extraction and of adaptive deformable convolution like [ 4 ] . Given the great performance obtained by diffusion models in image generation, in this work we consider adapting this type of models to the VFI problem. Other works have leveraged diffusion models in VFI such as [ 6 ] or [ 10 ] , but they share a characteristic coming from the original diffusion model: Their slowness due to the massive