3AM: 3egment Anything with Geometric Consistency in Videos
Yang-Che Sun 1 Cheng Sun 2 Chin-Yang Lin 1 Fu-En Yang 2 Min-Hung Chen 2 Yen-Yu Lin 1 Yu-Lun Liu 1 1 National Yang Ming Chiao Tung University 2 NVIDIA Research · 2026
Read 3AM: 3egment Anything with Geometric Consistency in Videos by Yang-Che Sun 1 Cheng Sun 2 Chin-Yang Lin 1 Fu-En Yang 2 Min-Hung Chen 2 Yen-Yu Lin 1 Yu-Lun Liu 1 1 National Yang Ming Chiao Tung University 2 NVIDIA Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08831.txt.
Introduction
Yang-Che Sun and a team from Taiwan and NVIDIA work on how to mark objects in video. Top tools can mark an object in each frame by use of past frames. But when the view shifts a lot, looks change and marks drift. Old 3D tools fix drift, yet they need much prep work and site data.
Their tool is called 3AM. It adds 3D aware traits from a model named MUSt3R to a top video tool named SAM2. The add on is used at train time, so run time stays light. A reader who marks video gets a path to firm marks through big view shifts. It needs no depth maps or pose data at run time.
Catalog record
| CREATOR | Yang-Che Sun 1 Cheng Sun 2 Chin-Yang Lin 1 Fu-En Yang 2 Min-Hung Chen 2 Yen-Yu Lin 1 Yu-Lun Liu 1 1 National Yang Ming Chiao Tung University 2 NVIDIA Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3AM: 3egment Anything with Geometric Consistency in Videos Yang-Che Sun 1 Cheng Sun 2 Chin-Yang Lin 1 Fu-En Yang 2 Min-Hung Chen 2 Yen-Yu Lin 1 Yu-Lun Liu 1 1 National Yang Ming Chiao Tung University 2 NVIDIA Research Abstract Video object segmentation methods like SAM2 achieve strong performance through memory-based architectures but struggle under large viewpoint changes due to reliance on appearance features. Traditional 3D instance segmentation methods address viewpoint consistency but require camera poses, depth maps, and expensive preprocessing. We introduce 3AM, a training-time enhancement that integrates 3D-aware features from MUSt3R into SAM2. Our lightweight Feature Merger fuses multi-level MUSt3R features that encode implicit geometric correspondence. Combined with SAM2’s appearance features, the model achieves geometry-consistent recognition grounded in both spatial position and visual similarity. We propose a field-of-view aware sampling strategy ensuring frames observe spatially consistent object regions for reliable 3D correspondence learning. Critically, our method requires only RGB input at inference, with no camera poses or preprocessing. On challenging datasets with wide-baseline motion (ScanNet++, Replica), 3AM substantially outperforms SAM2 and extensions, achieving 90.6% IoU and 71.7% Positive IoU on ScanNet++’s Selected Subset, improving over state-of-the-art VOS methods by +15.9 and +30.4 points. Project page: https://jayisaking.github.io/3AM-Page/ Figure 1 : Overview of 3AM. Given an input video or an unconstrained photo collection, 3AM takes a user-provided prompt, e.g., mask, point, or box, and produces a consistent object track across all views. Our method maintains cross-view correspondence even under large viewpoint changes, cluttered scenes, and variations in capture conditions, enabling robust object tracking from both videos and casual multi-view image sets. 1 Introduction Figure 2 : Limitations of traditional VOS and 3D segmentation approaches, and an overview of our capability. (a) Traditional VOS methods such as SAM2 Ravi et al. 2024 lose track when the camera undergoes large viewpoint changes, causing masks to drift or disappear. (b) 3D segmentation approaches rely on accurate camera poses and explicit 3D mask merging; they often propagate errors when the 3D reconstruction is incomplete or noisy. (c) Our 3AM consistently tracks object instances across drastic viewpoint changes without requiring camera poses or 3D ground-truth masks, demonstrating robust cross-view correspondence purely from geometry-aware 2D tracking. Video object segmentation (VOS) identifies and tracks target objects throughout a video sequence with consistent masks across frames. This is fundamental to autonomous driving, robotics, augmented reality, and video editing. A critical challenge is maintaining object identity under large viewpoint variations, temporary disappearances, or similar distractors. Traditional VOS methods rely on 2D appearance features and fail when the object appears dramatically different from varying viewpoints. For embodied AI in dynamic 3D environments, consistently recognizing objects across diverse viewpoints is essential for robust scene understanding. Recent video object segmentation advances follow two parallel lines. First, 2D foundation models with memory-based architectures: SAM2 Ravi et al. 2024 introduced promptable segmentation with streaming memory and spatio-temporal attention. SAM2Long Ding et al. 2024 employs memory trees for long-range identity preservation, while DAM4SAM Videnovic et al. 2025 incorporates distractor-aware updates. However, purely 2D approaches fail under large viewpoint changes as appearance features cannot establish reliable correspondence. Second, 3D instance segmentation methods: proposal-centric approaches like Mask3D Schult et al. 2022 and OneFormer3D Kolodiazhnyi et al. 2024 operate on point clouds, while projection-based methods like Open3DIS Nguyen et al. 2024 , SAM3D Yang et al. 2023 , and SAM2Object Zhao et al. 2025 lift 2D masks to 3D. These require camera poses, depth maps, preprocessing, and super-linear computational scaling. Both paradigms have critical gaps ( Fig. 2 ). 2D methods like SAM2 excel at efficiency but lack geometric awareness, failing under viewpoint variation ( Fig. 2 a). MOSEv2 Ding et al. 2025 shows significant degradation under wide-baseline changes. Conversely, 3D methods achieve better consistency but require explicit 3D inputs ( Fig. 2 b) and struggle to generalize. 2D-to-3D lifting suffers from view-inconsistent predictions Jia et al. 2024 ; Peng et al. 2023 ; Takmaz et al. 2023 ; Xu et al. 2024a ; Zhao et al. 2025 . End-to-end approaches like PanSt3R Zust et al. 2025 require offline processing. Can we achieve 3D-aware, viewpoint-robust segmentation while preserving promptability and efficiency, without explicit 3D supervision? ( Fig. 2 c) We introduce 3AM ( Fig. 1 ), a training-time enhancement to SAM2 that integrates 3D-aware features from reconstruction models. Our key insight is that MUSt3R Cabon et al. 2025 encodes implicit geometric correspondence through features learned from multi-view consistency. By fusing multi-level MUSt3R features with SAM2’s appearance features via a lightweight Feature Merger, we enable geometry-consistent recognition based on spatial position. This geometric awareness is learned from posed RGB video during training, but at inference requires only RGB input and prompts. We introduce a field-of-view aware sampling strategy ensuring frames observe spatially consistent object regions. On ScanNet++ Yeshwanth et al. 2023 and Replica Straub et al. 2019 with wide-baseline motion, 3AM achieves 90.6% IoU and 71.7% Positive IoU versus SAM2Long’s 74.7% and 41.3% (+15.9 and +30.4 points). In summary, our contributions are: • A key observation on the gap between 2D tracking and 3D consistency. 2D tracking is fundamentally limited under large viewpoint variations. Existing 3D approaches require camera poses, depth, or 3D masks, often unavailable in practice. This motivates our geometry-aware framework, which achieves cross-view consistency without 3D supervision at inference. • A 3D-aware feature integration framework fusing geometric features from 3D reconstruction models with SAM2’s appearance features, and a field-of-view aware training strategy ensuring geometric consistency for effective 3D correspondence learning. • Comprehensive experimental validation demonstrating substantial improvements over SAM2 on challenging datasets (ScanNet++, Replica), particularly on object reappearance and viewpoint changes. 2 Related Work 2D Video Object Segmentation. Video object segmentation (VOS) segments and tracks target objects across video sequences with temporally coherent masks Perazzi et al. 2016 ; Caelles et al. 2017 ; Voigtlaender et al. 2019 ; Maninis et al. 2018 ; Wu et al. 2025a . Early approaches relied on appearance propagation or optical flow but struggled with long-term consistency and occlusion Yang et al. 2019 ; Oh et al. 2019 ; Bhat et al. 2020 . Recent memory-based architectures leverage spatio-temporal attention and feature retrieval for improved object association Yang et al. 2021 ; Cheng and Schwing 2022 ; Cheng et al. 2021 ; Liu et al. 2022 ; Park et al. 2022 ; Xu et al. 2022 ; Yang et al. 2024b ; Yang and Yang 2022 ; Li et al. 2024a ; Cheng et al. 2024 . SAM2 Ravi et al. 2024 , a promptable model with streaming memory supporting points, boxes, and masks, achieves strong results across benchmarks. Follow-up efforts extend SAM2 Yang et al. 2024a ; Yang et al. 2025b ; Ye et al. 2025 ; Ding et al. 2024 ; Videnovic et al. 2025 : SAM2Long Ding et al. 2024 introduces memory trees for long-range identity preservation, while DAM4SAM Videnovic et al. 2025 incorporates distractor-aware updates for cluttered scenes. Despite these advances, large viewpoint changes, occlusion, and disappearance-reappearance events cause significant degradation, as shown by MOSEv2 Ding et al. 2025 ; Xu et al. 2018 ; Pont-Tuset et al. 2017 . To address these limitations, our method integrates 3D-aware representations into SAM2 for object-consistent segmentation under challenging viewpoint and appearance variations. 3D Instance Segmentation. Mainstream 3D instance