Training for X-Ray Vision: Amodal Segmentation, Amodal Content Completion, and View-Invariant Object Representation from Multi-Camera Video

Moore, Alexander [email protected] Saini, Amar [email protected] Cancilla, Kylie [email protected] Poland, Doug [email protected] Carrano, Carmen [email protected] · 2025

Read Training for X-Ray Vision: Amodal Segmentation, Amodal Content Completion, and View-Invariant Object Representation from Multi-Camera Video by Moore, Alexander [email protected] Saini, Amar [email protected] Cancilla, Kylie [email protected] Poland, Doug [email protected] Carrano, Carmen [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00339.txt.

Introduction

Alexander Moore and co-authors present a new data set for teaching computers to see whole objects even when parts are hidden. In busy scenes, one object often blocks another, and a model must guess the full shape. Their 2025 data adds a fresh angle: many cameras watch the same scene at once.

Simulated video shows cluttered rooms full of common household objects. Each object keeps the same tag across frames and across cameras, even as it moves. The set holds labels for about five point eight million object cases. It is the largest of its kind, and the first to give the true look of hidden parts. Others can use it to train and test sight models.

Catalog record

CREATORMoore, Alexander [email protected] Saini, Amar [email protected] Cancilla, Kylie [email protected] Poland, Doug [email protected] Carrano, Carmen [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training for X-Ray Vision: Amodal Segmentation, Amodal Content Completion, and View-Invariant Object Representation from Multi-Camera Video Moore, Alexander [email protected] Saini, Amar [email protected] Cancilla, Kylie [email protected] Poland, Doug [email protected] Carrano, Carmen [email protected] Abstract Amodal segmentation and amodal content completion require using object priors to estimate occluded masks and features of objects in complex scenes. Recent amodal segmentation work has introduced using temporal features to enrich object representations for improved amodal segmentation and enforce the concept of object permanence and temporal consistency in amodal video segmentation models which modal video object segmentation lacks. Until now, no data has provided an additional dimension for object context: the possibility of multiple cameras sharing a view of a scene. We introduce MOVi-MC-AC: M ultiple O bject Vi deo with M ulti- C ameras and A modal C ontent , the largest amodal segmentation and first amodal content dataset to date. Cluttered scenes of generic household objects are simulated in multi-camera video. MOVi-MC-AC contributes to the growing literature of object detection, tracking, and segmentation by including two new contributions to the deep learning for computer vision world. Multiple Camera (MC) settings where objects can be identified and tracked between various unique camera perspectives are rare in both synthetic and real-world video. We introduce a new complexity to synthetic video by providing consistent object ids for detections and segmentations between both frames and multiple cameras each with unique features and motion patterns on a single scene. Amodal Content (AC) is a reconstructive task in which models predict the appearance of target objects through occlusions. In the amodal segmentation literature, some datasets have been released with amodal detection, tracking, and segmentation labels. However, to date no dataset has provided ground-truth amodal content labels. While other methods rely on slow cut-and-paste schemes to generate amodal content pseudo-labels, they do not account for natural occlusions present in the modal masks. MOVi-MC-AC provides labels for 5.8 million object instances, setting a new maximum in the amodal dataset literature, along with being the first to provide ground-truth amodal content. The full dataset is available at https://huggingface.co/datasets/Amar-S/MOVi-MC-AC 1 Introduction 0 0 footnotetext: This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. Release number: LLNL-JRNL-2007808 1 1 footnotemark: 1 The ability to conceive of whole objects from glimpses at parts of objects is called gestalt psychology [ 1 ] . The shape and size of object bounding boxes and masks in video may rapidly change as objects undergo changes in position or occlusion through time. Tracking [ 2 , 3 ] , video object segmentation [ 4 , 5 , 6 ] , object retrieval and re-identification [ 7 , 8 ] , and video inpainting [ 9 , 10 ] could benefit from consistent object representations which maintain a cohesive object view invariant of occlusion, representation, or perspective change [ 11 ] . Amodal segmentation and content completion are vital in real-world applications of machine learning requiring consistent object understanding and object permanence through complex video such as robots and autonomous driving [ 12 , 13 , 14 ] . Monocular image amodal segmentation models [ 15 , 16 , 17 , 18 , 19 ] rely on object priors to estimate occluded object size and shape through obscurations. Recent monocular video amodal segmentation models [ 20 , 21 ] use context from temporally distant video features to estimate amodal segmentation masks across time by exploiting object priors of video diffusion models trained on synthetic data. So far, no existing research has investigated using multi-view images and video to generate consistent object representations for the purpose of amodal segmentation. We further develop this research area to introduce multi-view video amodal content completion, a new task in which object visuals are estimated through occlusion using both temporal context as well as multi-view information. We release the first dataset to contain ground-truth amodal segmentation masks for all objects in the scene as well as ground-truth amodal content (or the visible ”x-ray” view) of all objects in every scene. MOVi-MC-AC (Ours) MOVI-Amodal (Amazon) SAIL-VOS 3D SAIL-VOS COCOA COCOA-cls D2S DYCE Statistics Image or Video Video Video Video Video Image Image Image Image Synthetic or Real Synthetic Synthetic Synthetic Synthetic Real Real Real Synthetic Number of Video Scenes 2041 838 203 201 - - - - Number of Scene Images 293,904 20,112 237,611 111,654 5,073 3,499 5,600 5,500 Number of Classes 1,033 930 178 162 - 80 60 79 Number of Instances 5,899,104 295,176 3,460,213 1,896,296 46,314 10,562 28,720 85,975 Number of Occluded Instances 4,089,229 247,565 - 1,653,980 28,106 5,175 16,337 70,766 Average Occlusion Rate 45.2% 52.0% - 56.3% 18.8% 10.7% 15.0% 27.7% Provided Modalities Scene-Level RGB Frames Yes Yes Yes Yes Yes Yes Yes Yes Modal Object Masks Yes Yes Yes Yes Yes Yes Yes Yes Model Object RGB Content Yes Yes Yes Yes Yes Yes Yes Yes Scene-Level (Modal) Depth Masks Yes Yes Yes Yes No No No No Amodal Object Masks Yes Yes Yes Yes Yes Yes Yes Yes Amodal Object RGB Content Yes No No No No No No No Amodal Object Depth Masks Yes No No No No No No No Multiple Cameras (multi-view) Yes No No No No No No No Scene-object descriptors (instance re-id) Yes Yes No No No No No No Table 1 : A comparison of contemporary datasets for amodal video object segmentation, tracking, and amodal content completion. While MOVi-MC-AC does not contain articulating objects, it is by far the largest dataset which provides complete modal masks, amodal masks, and amodal content for all objects in every scene as well as six distinct camera views with unique camera extrinsics and motion patterns with a united object identification label enabling new directions of research in computer vision. (a) Camera 1. (b) Camera 6. Figure 1 : Amodal content completion from multiple cameras must leverage temporal information from one camera view as well as multiple camera perspectives to most accurately predict the visual features of highly-occluded objects from many perspectives simultaneously. MOVi-MC-AC is the first dataset to include ground truth amodal content of occluded objects in video as well as the option to utilize information from multiple cameras on the same scene. Amodal video segmentation and amodal content completion models share information between temporal and camera features to estimate the unoccluded view of objects. Rows: (1) RGB, (2) Modal/Amodal Masks, (3) Amodal Content (4) Overlay. Each column is a new frame in the video. (a) Up to six cameras observe the same scene with different perspectives and motion characteristics. (b) Object visible features are partially observed by each camera from one perspective. (c) Amodal object content is the ground-truth unobscured view of the object generated. Figure 2 : Amodal content completion from multiple cameras must leverage temporal information from one camera view as well as multiple camera perspectives to most accurately predict the visual features of highly-occluded objects from many perspectives simultaneously. MOVi-MC-AC is the first dataset to include ground truth amodal content of occluded objects in video as well as the option to utilize information from multiple cameras on the same scene. Amodal video segmentation and amodal content completion models share information between temporal and camera features

Read the complete text