MatAnyone: Stable Video Matting with Consistent Memory Propagation

Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, Chen Change Loy · 2025

Read MatAnyone: Stable Video Matting with Consistent Memory Propagation by Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, Chen Change Loy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14677.txt.

Introduction

Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, and Chen Change Loy wrote this computer vision paper, published in two thousand twenty-five, on cutting a person out of video cleanly. Video matting finds the fine edge of hair and motion frame by frame, and methods that use each frame alone wobble when backgrounds get messy.

Their system, MatAnyone, lets the user assign the target once, then carries a memory of the subject forward. A region-adaptive fusion blends the last frame's memory so core areas stay stable while edges keep their detail. The team also built a larger, more varied training set for the task. A training scheme that borrows big segmentation data adds steadiness. On real-world clips, the results beat prior methods in both accuracy and calm over time.

Catalog record

CREATORPeiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, Chen Change Loy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MatAnyone: Stable Video Matting with Consistent Memory Propagation Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, Chen Change Loy Abstract Auxiliary-free human video matting methods, which rely solely on input frames, often struggle with complex or ambiguous backgrounds. To address this, we propose MatAnyone, a robust framework tailored for target-assigned video matting. Specifically, building on a memory-based paradigm, we introduce a consistent memory propagation module via region-adaptive memory fusion, which adaptively integrates memory from the previous frame. This ensures semantic stability in core regions while preserving fine-grained details along object boundaries. For robust training, we present a larger, high-quality, and diverse dataset for video matting. Additionally, we incorporate a novel training strategy that efficiently leverages large-scale segmentation data, boosting matting stability. With this new network design, dataset, and training strategy, MatAnyone delivers robust and accurate video matting results in diverse real-world scenarios, outperforming existing methods. Figure 1 : Our MatAnyone is capable of producing highly detailed and temporally consistent alpha mattes throughout a video. (a) It adapts to a variety of frame sizes and media types ( e.g . , films, games, smartphone videos), achieving fine-grained details at the image-matting level. (b) RVM Lin et al. 2022 , an auxiliary-free video matting method, struggles with complex or ambiguous backgrounds. In contrast, our method effectively isolates the target object from such distractors, preserving a clean background and complete foreground parts. (c) Our method also excels at consistently tracking the target ( i.e . , the lady in pink) even in scenes containing multiple salient objects ( i.e . , the man and the lady). It accurately distinguishes between them even during their interactions. (Zoom-in for best view) 1 Introduction Figure 2 : Definitions and motivations for MatAnyone. (a) In a matting frame, the image can be broadly divided into two areas based on the alpha value: the core (semantic) and the boundary (fine-details). The core includes the background (alpha values of 0) and the solid foreground (alpha values of 1), while the boundary (highlighted in pink) encompasses areas with alpha values between 0 and 1. (b) Due to the under-defined setting, auxiliary-free methods like RVM Lin et al. 2022 are easily confused by ambiguous background. Meanwhile, mask-guided methods like MaGGIe Huynh et al. 2024 tend to break the segmentation prior they aim to leverage, due to the deficiency in video matting data. Auxiliary-free human video matting (VM) is widely recognized for its convenience Lin et al. 2022 ; Ke et al. 2022b ; Li et al. 2024a , as it only requires input frames without additional annotations. However, its performance often deteriorates in complex or ambiguous backgrounds, especially when similar objects, i.e . , other humans, appear in the background (Fig. 2 (b)). We consider auxiliary-free video matting to be under-defined, as their results can be uncertain without a clear target object. In this work, we focus on a problem that is more applicable to real-world video applications: video matting focused on pre-assigned target object(s), with the target segmentation mask provided in the first frame. This enables the model to perform stable matting via consistent object tracking throughout the entire video, while offering better interactivity. The setting is well-studied in Video Object Segmentation (VOS), where it is referred to as “semi-supervised” Hu et al. 2017 ; Oh et al. 2019b ; Cheng et al. 2021b . A common strategy is to use a memory-based paradigm Oh et al. 2019b ; Xie et al. 2021 ; Cheng and Schwing 2022 ; Cheng et al. 2024 , encoding past frames and corresponding segmentation results into memory, from which a new frame retrieves relevant information for its mask prediction. This allows a lightweight network to achieve consistent and accurate tracking of the target object. Inspired by this, we adapt the memory-based paradigm for video matting, leveraging its stability across frames. Video matting poses additional challenges compared to VOS, as it requires not only accurate semantic detection in core regions but also high-quality detail extraction along the boundary ( e.g . , hair), as defined in Fig. 2 (a). A straightforward approach is to fine-tune matting details using matting data, based on segmentation priors from VOS. Recent approaches attempt to achieve both goals, either in a coupled or decoupled manner. For instance, AdaM Lin et al. 2023 and FTP-VM Huang and Lee 2023b refine the memory-based segmentation mask for each frame via a decoder to produce alpha mattes, while MaGGIe Huynh et al. 2024 devises a separate refiner network to process segmentation masks across all frames from an off-the-shelf VOS model. However, these methods often lead to suboptimal results due to limitations in the available video matting data: (i) the quality of VideoMatte240K Lin et al. 2021 , the most widely used human video matting dataset, is suboptimal. Its ground-truth alpha mattes exhibit problematic semantic accuracy in core areas ( e.g . , interior holes) and lack fine details along the boundaries ( e.g . , blurry hair); (ii) video matting datasets are much smaller in scale compared to VOS datasets; and (iii) video matting data are synthetic due to the extreme difficulty of human annotations, limiting their generalizability to real-world cases Lin et al. 2022 . Consequently, fine-tuning a strong VOS prior for video matting with existing video matting data usually disrupts this prior. While boundary details may show improvement compared to segmentation results, the matting quality in terms of semantic stability in core areas and details in boundary areas remain unsatisfactory, as shown by the results of MaGGIe in Fig. 2 (b). Producing matting-level details while maintaining semantic stability of a memory-based approach is challenging, especially training with suboptimal video matting data. To tackle this, we focus on several key aspects: Network - we introduce a consistent memory propagation mechanism in the memory space. For each current frame, the alpha value change relative to the previous frame is estimated for every token. This estimation guides the adaptive integration of information from the previous frame. The “large-change” regions rely more on the current frame’s information queried from the memory bank, while “small-change” regions tend to retain the memory from the previous frame. This region-adaptive memory fusion inherently stabilizes memory propagation throughout the video, improving matting quality with fine details and temporal consistency. Specifically, it encourages the network to focus on boundary regions during training to capture fine details, while “small-change” tokens in the core regions preserve internally complete foreground and clean background (see our results in Fig. 2 (b)). Data - we collect a new training dataset, named VM800 , which is twice as large, more diverse, and of higher quality in both core and boundary regions compared to the VideoMatte240K dataset Lin et al. 2021 , greatly enhancing robust training for video matting. In addition, we introduce a more challenging test dataset, named YoutubeMatte , featuring more diverse foreground videos and improved detail quality. These new datasets offer a solid foundation for robust training and reliable evaluation in video matting. Training Strategy - the lack of real video matting data remains a significant limitation, affecting both stability and generalizability. We address this problem by leveraging large-scale real segmentation data via a novel training strategy. Unlike common practices Lin et al. 2022 ; Huynh

Read the complete text