Automated Video Segmentation Machine Learning Pipeline Conference: The Digital Production Symposium; August 9, 2025; Vancouver, BC, Canada The Digital Production Symposium (DigiPro ’25), August 9, 2025, Vancouver, BC, Canada DOI: 10.1145/3744199.3744635 ISBN: 979-8-4007-2008-6/2025/08 3580 CCS: Comp

Johannes Merz, Lucien Fostier · 2025

Read Automated Video Segmentation Machine Learning Pipeline Conference: The Digital Production Symposium; August 9, 2025; Vancouver, BC, Canada The Digital Production Symposium (DigiPro ’25), August 9, 2025, Vancouver, BC, Canada DOI: 10.1145/3744199.3744635 ISBN: 979-8-4007-2008-6/2025/08 3580 CCS: Comp by Johannes Merz, Lucien Fostier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07242.txt.

Introduction

Johannes Merz and Lucien Fostier automate the mask-cutting grind of visual effects. Their 2025 paper presents a machine learning pipeline that produces temporally steady instance masks for video, the outlines artists need before compositing new elements into a shot.

The pipeline chains three learned stages. Text prompts drive flexible object detection, so artists name what to cut out. Per-frame segmentation refines the outlines. Video tracking holds them consistent across frames, where hand work usually wobbles. Packaged in containers with a structured output format, the system was adopted quickly by the studio's artists. It cuts manual effort, speeds early composites, and hands over complete segmentation data for later stages, the authors report.

Catalog record

CREATORJohannes Merz, Lucien Fostier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07242

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Video Segmentation Machine Learning Pipeline Conference: The Digital Production Symposium; August 9, 2025; Vancouver, BC, Canada The Digital Production Symposium (DigiPro ’25), August 9, 2025, Vancouver, BC, Canada DOI: 10.1145/3744199.3744635 ISBN: 979-8-4007-2008-6/2025/08 3580 CCS: Computing methodologies Neural networks CCS: Computing methodologies Image manipulation Johannes Merz, Lucien Fostier Abstract . Visual effects (VFX) production often struggles with slow, resource-intensive mask generation. This paper presents an automated video segmentation pipeline that creates temporally consistent instance masks. It employs machine learning for: (1) flexible object detection via text prompts, (2) refined per-frame image segmentation and (3) robust video tracking to ensure temporal stability. Deployed using containerization and leveraging a structured output format, the pipeline was quickly adopted by our artists. It significantly reduces manual effort, speeds up the creation of preliminary composites, and provides comprehensive segmentation data, thereby enhancing overall VFX production efficiency. Figure 1 . Mask output from a shot in Dune: Prophecy , generated by our automatic video segmentation pipeline and stored in our custom ObjectId file format. © HBO 1. Introduction Modern visual effects (VFX) production is characterized by tight turnaround times and increasingly animation-intensive workflows. Maintaining high quality under these pressures necessitates initiating animation work and delivering preview composites as early as possible in the schedule. These preview composites, which integrate computer graphic elements roughly composited onto background image plates, require masks to isolate foreground elements. Generating these essential masks using existing traditional techniques, such as manual rotoscopy or bluescreen cleanup, proves notoriously time-consuming. Challenges like uneven bluescreens frequently necessitate significant manual effort to prepare masks. Importantly, much of this work is dedicated to producing assets that are ultimately disposable, serving only the immediate needs of temporary reviews or downstream department handovers. This results in a considerable waste of skilled resources on tasks that are only required for transient purpose, as these temporary masks often cannot be directly reused for final quality work due to differing requirements. These significant production challenges underscore a clear and pressing need for a more efficient, automated approach to mask generation in early-stage VFX workflows. 2. Automatic Segmentation Pipeline Our main goal is to create a system that enables downstream departments to quickly pick up their respective tasks while minimizing the delay as much as possible. To achieve this, we developed a pipeline that can be run fully automatically without needing any user interaction. The default behaviour of this pipeline is designed to create an individual mask for each person in a plate, while being temporally consistent. The pipeline can be scheduled to run automatically when plates are ingested. Optionally, the default behaviour can be modified so that the pipeline generates masks for other objects and can then be run manually. After running the pipeline, the output is a temporally consistent instance segmentation. In general segmentation can be classified into semantic and instance segmentation. Semantic segmentation generates a mask that identifies all areas in an image that belong to the same class (e.g. people), but cannot differentiate between individual instances. This allows to e.g. replace backgrounds. Instance segmentation is a harder problem that can also differentiate between the individual instances in that image (e.g. identify individual people) and enables artists to place objects between characters. The pipeline consists of three fundamental stages as visualized in Figure 2 : (1) Object Detection (2) Image Segmentation (3) Video Tracking Figure 2 . Overview of automatic video segmentation (see Section 2 ) and integration into our VFX pipeline (see Section 4 ). The following sections will describe each stage. Without limiting the generality we will assume that the pipeline is segmenting people for the sake of simplicity of the description. 2.1. Object Detection - Stage 1 The first step in the pipeline performs object detection. This is the process of detecting all instances of a specific object in a shot. In machine learning (ML) multiple approaches for object detection exist. One type of neural network can be trained on specific categories (closed-set detection), which allows it to detect these categories with high accuracy. A popular system of this type is You Only Look Once (YoLo) ( Redmon et al. 2016 ) , which is pretrained on some categories, including “person” . However, while testing YoLo on our shots, we quickly discovered that its limitations make it not a good fit for our use case in particular. Although it can detect people in general, the huge variety of possible human-like appearances in VFX can pose a challenge. For example, a character wearing a futuristic space helmet could potentially stump a pretrained base model due to the lack of similar samples in the public dataset the model has been trained on. Moreover, in VFX we mostly want to predict people, but not always. In order to modify the behaviour we would need to create a new dataset for each instance and thus significantly increase the manual work required. To give the pipeline more flexibility without requiring data collection and fine-tuning for each category, we chose an open-set detection system instead, which generates a prediction given a conditioning input. Specifically, we chose to use GroundingDINO ( Liu et al. 2024 ) , which can be prompted using natural language. With the help of this object detection model, the artists can prompt the pipeline to generate masks for any object in the shot, simply by typing it out. Additionally, prompts can be concatenated which allows for the detection of different objects at the same time. For example a prompt could be "person, car" to instruct the pipeline to generate individual masks for each person and each car across the whole shot. When our pipeline is run in the default automatic mode, the prompt “person” is used. In this first step of the pipeline, the object detection is performed on each individual frame of the shot and outputs bounding boxes for each instance of the prompted objects. The text prompt "person" has proven to detect any human-like appearance with high probability and can therefore run unsupervised for the default case. Mask generation of other objects requires artists to manually edit the prompt. Since the pipeline tasks are dispatched to our render farm the interaction is not in real time, making it unfeasible to experiment with the text prompt by waiting for the final results. To solve this problem, we created a simple browser-based application to serve as a test bench for the artists (Figure 3 ) that provides an easy visualization of the bounding boxes. It also previews the results of stage 2, which is described in detail in Section 2.2 . Figure 3 . Object detection preview tool to quickly experiment with text prompts for stage 1 of the pipeline. It also shows a preview of the segmentation post-processing performed in stage 2. Images © HBO 2.2. Image Segmentation - Stage 2 The detections from the previous stage are a good first step for our pipeline, yet there are many potential issues that we need to address on a per-frame basis since the object detection is not guaranteed to be without fault: (1) Sometimes the same person might be detected multiple times in a frame, or parts of the same person might be detected in separate channels. (2) Especially in

Read the complete text