VideoMaMa: Mask-Guided Video Matting via Generative Prior
Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, [0.5em] Seungryong Kim, Joon-Young Lee, [0.8em] Korea University · 2026
Read VideoMaMa: Mask-Guided Video Matting via Generative Prior by Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, [0.5em] Seungryong Kim, Joon-Young Lee, [0.8em] Korea University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14255.txt.
Introduction
Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, and their co-authors present VideoMaMa in this 2026 paper. The topic is video matting, the task of cutting a subject out of each frame with soft, exact edges. The output is an alpha matte, a map of how clear each pixel is. Good matting models are hard to train for real-world clips.
The main block is a lack of labeled data. VideoMaMa starts from coarse segmentation masks, rough maps of where the subject is. It turns them into pixel-exact mattes. To do so, it draws on pretrained video diffusion models, which bring strong prior knowledge of how real scenes look and move.
Catalog record
| CREATOR | Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, [0.5em] Seungryong Kim, Joon-Young Lee, [0.8em] Korea University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14255
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoMaMa: Mask-Guided Video Matting via Generative Prior Sangbeom Lim, Seoung Wug Oh, Jiahui Huang, Heeji Yoon, [0.5em] Seungryong Kim, Joon-Young Lee, [0.8em] Korea University Abstract Generalizing video matting models to real-world videos remains a significant challenge due to the scarcity of labeled data. To address this, we present Video Mask-to-Matte Model ( VideoMaMa ) that converts coarse segmentation masks into pixel accurate alpha mattes, by leveraging pretrained video diffusion models. VideoMaMa demonstrates strong zero-shot generalization to real-world footage, even though it is trained solely on synthetic data. Building on this capability, we develop a scalable pseudo-labeling pipeline for large-scale video matting and construct the Matting Anything in Video ( MA-V ) dataset, which offers high-quality matting annotations for more than 50K real-world videos spanning diverse scenes and motions. To validate the effectiveness of this dataset, we fine-tune the SAM2 model on MA-V to obtain SAM2-Matte, which outperforms the same model trained on existing matting datasets in terms of robustness on in-the-wild videos. These findings emphasize the importance of large-scale pseudo-labeled video matting and showcase how generative priors and accessible segmentation cues can drive scalable progress in video matting research. Figure 1 : We introduce Video Mask-to-Matte Model ( VideoMaMa ), a diffusion-based model that generates high-quality alpha mattes from input binary segmentation masks obtained either from existing models such as SAM2 Ravi et al. 2024 or from ground-truth segmentation masks in existing datasets such as SA-V Ravi et al. 2024 . Examples shown highlights our VideoMaMa’s ability to capture fine-grained details including motion blur, and intricate boundary structures on natural video footage. 2 2 footnotetext: Work done during an internship at Adobe Research. 1 Introduction Video matting, the task of extracting foreground objects with pixel-level precision from video, serves as a fundamental component in video editing applications, including background replacement Zeng et al. 2025 ; Gao et al. 2025 ; Ren et al. 2024 , visual composition Dalva et al. 2024 ; Yang et al. 2025a , and relighting Chaturvedi et al. 2025 ; Ren et al. 2024 ; Wang et al. 2025a . Despite its importance, training a video matting model that performs robustly across diverse real-world videos remains challenging for two key reasons. First, high-quality video matting annotations are extremely scarce. Ground-truth mattes are typically captured in controlled environments such as chroma key studios Lin et al. 2021 ; Zhang et al. 2021 or with specialized camera setups Enomoto et al. 2025 , which makes it difficult to scale annotation efforts and to capture diverse objects, scenes, and camera angles. As a result, existing datasets are often limited to human portraits. Second, most video matting models are trained on synthetic videos, where foregrounds are composited onto arbitrary backgrounds. Such synthetic compositions often introduce unrealistic artifacts in lighting, motion blur, and temporal coherence. These two factors, limited data diversity and the domain gap between synthetic and real videos, significantly hinder the ability of current models to generalize to real-world footage where complex interactions between foregrounds and backgrounds naturally occur. In this work, we propose a novel bootstrapping strategy that effectively bridges the gap between synthetic and real-world videos, enabling more robust video matting. We introduce Video Mask-to-Matte Model ( VideoMaMa ), a diffusion-based model that converts binary segmentation masks into continuous alpha mattes (see Figure 1 ). Built upon pretrained video diffusion models Blattmann et al. 2023 , VideoMaMa demonstrates remarkable zero-shot generalization to real-world videos, even though the model is trained exclusively on synthetic data. This strong generalization ability stems from diffusion priors trained on internet-scale image and video data, which can generate high-quality content across diverse domains HaCohen et al. 2024 ; Wan et al. 2025 ; Blattmann et al. 2023 . To adapt these priors for video matting while preserving their generative capabilities, we introduce a two-stage training strategy that separately optimizes spatial and temporal layers, along with semantic knowledge injection via DINOv3 Siméoni et al. 2025 features. To evaluate the robustness of the VideoMaMa model, we conduct experiments on video matting guided by binary masks from diverse sources, including those produced by existing video segmentation models. The results show that VideoMaMa consistently generates high-quality video matting outputs regardless of the input mask type, demonstrating strong robustness. These findings suggest that VideoMaMa can be effectively utilized as a pseudo-labeler for constructing large-scale video matting annotations. In addition, by leveraging VideoMaMa, we propose a simple but scalable pipeline for constructing large-scale video matting annotations and introduce the Matting Anything in Videos ( MA-V ) dataset, the first large-scale pseudo video matting dataset built by converting segmentation labels from the SA-V dataset Ravi et al. 2024 . Unlike existing video matting datasets Huynh et al. 2024 ; Ge et al. 2025 ; Yang et al. 2025b ; Zhang et al. 2021 ; Lin et al. 2021 , which rely on synthetic or composited content, MA-V provides high-quality matting annotations for over 50K real-world videos covering diverse scenes, objects, and motion dynamics, as summarized in Table 1 . This scalable pipeline demonstrates the potential of leveraging generative priors and segmentation labels, which are considerably easier to obtain than video alpha matte, to efficiently construct large-scale, high-quality video matting datasets. To validate the effectiveness of large-scale pseudo video matting annotations, we fine-tune the SAM2 Ravi et al. 2024 model for video matting using MA-V , referred to as SAM2-Matte. Trained without any architectural modifications, the SAM2-Matte model achieves substantially more robust matting performance on in-the-wild videos than the same SAM2 model fine-tuned on existing video matting datasets, as well as other existing video matting methods. The results demonstrate the strong potential of large-scale pseudo annotations to drive advancements in video matting research. Our contributions can be summarized as follows. • VideoMaMa : A diffusion-based model that generates realistic video matting annotations from binary masks, enabling scalable matting label creation with easily obtainable segmentation labels. • MA-V Dataset : The first large-scale, high-quality video matting dataset built on real captured footage, consisting of over 50K diverse videos. 2 Related Work Video matting. Video matting methods can be categorized into auxiliary-free and auxiliary-guided approaches. Auxiliary-free methods Lin et al. 2022 ; Ke et al. 2022 ; Li et al. 2024 typically focus on portrait matting, restricting their applicability to human-centric scenarios. Trimap-guided approaches Huang and Lee 2023 ; Seong et al. 2022 require manual trimap annotations, limiting their practicality for zero-shot inference. Recent mask-based guidance methods address these limitations: MaGGIe Huynh et al. 2024 decouples tracking from matting using binary mask tracks, MatAnyone Yang et al. 2025b propagates target-assigned matting through a memory-augmented architecture, and GVM Ge et al. 2025 employs diffusion models for portrait matting. Despite these advances, existing methods remain constrained by domain specificity. Beyond algorithmic limitations, existing datasets also constrain progress. Representative datasets include VideoMatte240K Lin et al. 2021 (484 videos), VideoMatting108 Zhang et al. 2021 (108 videos), and VM800 Yang et al. 2025b (826 videos), predominantly focusing on human subjects in controlled settings Sun et al. 2021 ; Wang et al. 2021 ; Ge et al. 2025 . Critically, all rely on composition-based generation where foreground objects are composited onto random backgrounds, creating artificial scene compositions. No large-scale real video matting dataset exists where foreground and background naturally