OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models Journal: TOG Volume: 00 0 000 8 papers_1753
Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari · 2025
Read OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models Journal: TOG Volume: 00 0 000 8 papers_1753 by Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18033.txt.
Introduction
Dvir Samuel and colleagues present OmnimatteZero, a training-free method that splits video into layers. Their 2025 arXiv paper uses pre-trained video diffusion models. Each layer holds the background or one object. Shadows and reflections travel with their objects. Older methods needed heavy training or costly optimization.
Zero-shot image inpainting tricks fail on video by themselves. New guidance modules steer attention across space and time. They remove objects and rebuild consistent backgrounds. Self-attention maps reveal where objects leave their footprints. Extracted layers can be pasted onto fresh videos. The approach skips training entirely while keeping quality. Editors gain fast object removal and compositing tools.
Catalog record
| CREATOR | Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models Journal: TOG Volume: 00 0 000 8 papers_1753 Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari Abstract . In Omnimatte, one aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero , a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. These are accomplished by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. To overcome this, we introduce temporal and spatial attention guidance modules that steer the diffusion process for accurate object removal and temporally consistent background reconstruction. We further show that self-attention maps capture information about the object and its footprints and use them to inpaint the object’s effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime. Project Page. Figure 1 . OmnimatteZero is the first training-free generative approach for Omnimatte, leveraging pre-trained video diffusion models to achieve object removal, extraction, and seamless layer compositions in just 0.04 sec/frame (on an A100 GPU). See video examples in the supplemental material. <short description> <long description> 1. Introduction Extracting or isolating objects in videos is a central problem in video understanding and editing, with important applications such as removing, modifying, or enhancing elements within a video scene. One important flavor of this task is video matting ( Chuang et al. 2002 ) , which involves decomposing a video into a background layer and one or more foreground layers, each representing an individual object. A particularly challenging form of video matting is Omnimatte ( Wang and Adelson 1994 ; Lu et al. 2021 ) , where each foreground object is not only isolated but also includes associated effects such as shadows, reflections, and scene perturbations. The layers extracted by video matting facilitate various video editing applications, including object removal, background replacement, retiming ( Lu et al. 2020 ) , and object duplication ( Lee et al. 2024 ) , many of which require estimating occluded background regions using techniques like video inpainting or omnimatting. However, despite their promise, existing methods for video inpainting and Omnimatte face significant limitations. Recent inpainting approaches ( Li et al. 2025 ; Zhou et al. 2023 ) , while trained on annotated datasets for object removal, fail to account for the nuanced effects that objects impart on the scene. Similarly, current Omnimatte methods demand computationally intensive self-supervised optimization for each video ( Lu et al. 2021 ; Suhail et al. 2023 ; Lin et al. 2023 ; Lee et al. 2024 ) . Some approaches further rely on large hand-crafted training datasets ( Winter et al. 2024 ; Lee et al. 2024 ) or require 3D scene modeling using meshes or NeRFs ( Suhail et al. 2023 ; Lin et al. 2023 ) , resulting in hours of training and slow per-frame rendering. Given the success of fast, training-free approaches in image editing ( Lugmayr et al. 2022 ; Corneanu et al. 2024 ) , a natural question arises: can these methods be effectively adapted to video editing? In this paper, we present a new approach called OmnimatteZero , the first training-free generative method to Omnimatte. OmnimatteZero (named for its zero training and optimization requirements) leverages off-the-shelf pre-trained video diffusion models to remove all objects from a video, extract individual object layers along with their associated effects, and finally composite them onto new backgrounds. This entire process is performed efficiently during diffusion inference, making it significantly faster than current methods. See Figure 1 for illustration. We begin by adapting training-free image inpainting methods ( Lugmayr et al. 2022 ; Meng et al. 2022 ) to handle object inpainting in videos using diffusion models. However, we find that simply applying these frame-by-frame techniques to videos leads to poor results: missing regions are not filled convincingly, and temporal consistency breaks down. To address this, we propose two attention guidance modules that can be easily integrated into pre-trained video diffusion models. These modules guide the diffusion process to achieve both accurate object removal and consistent background reconstruction. The first module, Temporal Attention Guidance (TAG) , uses background patches from nearby frames. By modifying self-attention layers, it encourages the model to use these patches to reconstruct masked regions, improving temporal coherence. The second module, Spatial Attention Guidance (SAG) , operates within each frame to refine local details and enhance visual plausibility. We also observe that self-attention maps contain cues not just about the object but also its indirect effects, such as shadows and reflections. By leveraging this information, our method can inpaint these subtle traces, yielding cleaner backgrounds. For foreground layer extraction, we find that the object (and its traces) can be isolated by subtracting the latent representation of the clean background from the original video latent. This extracted object layer can then be seamlessly inserted into new backgrounds by simply adding the latents, enabling fast and flexible video composition. To validate our approach, we apply it to two video diffusion models, LTXVideo ( HaCohen et al. 2024 ) and Wan2.1 ( video 2025 ) , and evaluate it on standard Omnimatte benchmarks both quantitatively and qualitatively. Our method outperforms all supervised and self-supervised existing inpainting and Omnimatte techniques across all benchmarks in terms of background reconstruction and object layer extraction. Notably, OmnimatteZero requires no training or additional optimization, functioning entirely within the denoising process of diffusion models. This positions OmnimatteZero as the fastest Omnimatte technique to date. 2. Related Work 2.1. Omnimatte The first Omnimatte approaches ( Kasten et al. 2021 ; Lu et al. 2021 ) use motion cues to decompose a video into RGBA matte layers, assuming a static background with planar homographies. Optimization isolates dynamic foreground layers, taking 3 to 8.5 hours per video and \sim 2.5 seconds per frame to render. Later methods enhanced Omnimatte with deep image priors ( Lu et al. 2022 ; Gu et al. 2023 ) and 3D representations ( Lin et al. 2023 ; Suhail et al. 2023 ; Wu et al. 2024 ) , but still rely on strict motion assumptions and require extensive training ( \sim 6 hours) with rendering times of 2.5-3.5 seconds per frame. A recent generative approach ( Lee et al. 2024 ) uses video diffusion priors to remove objects and effects, with test-time optimization for frame reconstruction, taking \sim 9 seconds per frame. In this paper, we propose a novel, training and optimization-free method using off-the-shelf video diffusion models, achieving real-time performance in 0.04 seconds per frame. 2.2. Video Inpainting Earlier video inpainting methods ( Chang et al. 2019 ; Wang et al. 2019 ; Hu et al. 2020 )