T-3DGS: Removing Transient Objects for 3D Scene Reconstruction
Alexander Markin, Vadim Pryadilshchikov, Artem Komarichev, Ruslan Rakhimov, Peter Wonka, Evgeny Burnaev · 2024
Read T-3DGS: Removing Transient Objects for 3D Scene Reconstruction by Alexander Markin, Vadim Pryadilshchikov, Artem Komarichev, Ruslan Rakhimov, Peter Wonka, Evgeny Burnaev (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00155.txt.
Introduction
Alexander Markin and co-authors present T-3DGS in a 2024 arXiv paper. Moving objects in video, such as people or cars, can spoil 3D scene reconstructions. Their framework filters out such transient objects during reconstruction with Gaussian Splatting.
The first step uses an unsupervised network that spots transient items by their odd training behavior. The second step refines those spots with a segmentation tool and a two-way tracking module. This sharpens borders and keeps detection steady over time. Tests on sparse and dense video sets beat leading methods. The result is cleaner 3D scenes in hard, real-world footage.
Catalog record
| CREATOR | Alexander Markin, Vadim Pryadilshchikov, Artem Komarichev, Ruslan Rakhimov, Peter Wonka, Evgeny Burnaev |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00155
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
T-3DGS: Removing Transient Objects for 3D Scene Reconstruction Alexander Markin, Vadim Pryadilshchikov, Artem Komarichev, Ruslan Rakhimov, Peter Wonka, Evgeny Burnaev Abstract Transient objects in video sequences can significantly degrade the quality of 3D scene reconstructions. To address this challenge, we propose T-3DGS, a novel framework that robustly filters out transient distractors during 3D reconstruction using Gaussian Splatting. Our framework consists of two steps. First, we employ an unsupervised classification network that distinguishes transient objects from static scene elements by leveraging their distinct training dynamics within the reconstruction process. Second, we refine these initial detections by integrating an off-the-shelf segmentation method with a bidirectional tracking module, which together enhance boundary accuracy and temporal coherence. Evaluations on both sparsely and densely captured video datasets demonstrate that T-3DGS significantly outperforms state-of-the-art approaches, enabling high-fidelity 3D reconstructions in challenging, real-world scenarios. More results and code are available at https://transient-3dgs.github.io/ Figure 1 : Existing state-of-the-art methods, such as SpotLessSplats [ 22 ] , often struggle to correctly identify transient and semi-transient objects, leading to artifacts in 3D scene reconstruction. Our proposed T-3DGS method accurately detects all transient distractors, generates clean masks, and propagates them across frames. By effectively masking transient objects, T-3DGS enables high-fidelity novel view synthesis and significantly improves reconstruction quality from real-world image and video sequences. 2 2 footnotetext: Correspondence to: [email protected]. 1 Introduction Novel view synthesis and 3D scene reconstruction from multiple 2D images or videos are critical, rapidly evolving areas in computer vision. Neural Radiance Fields (NeRF) [ 15 ] and 3D Gaussian Splatting (3DGS) [ 9 ] have shown remarkable improvements in novel view synthesis on complex scenes. NeRF implicitly represents the scene as a volumetric function, and 3DGS explicitly represents it as a set of 3D Gaussians. Both approaches produce high-quality realistic images. There are multiple follow-up works for diverse downstream applications, including 3D scene reconstruction [ 29 , 13 , 8 ] , 3D synthesis [ 17 , 31 , 26 ] , semantic and language integration into 3D representations [ 25 , 10 , 24 ] . Both 3D Gaussian Splatting and NeRF optimize 3D scene reconstruction using photometric losses. High-quality results are achieved under the assumption that the captured scene is completely static and does not include any distractors , such as moving objects (i.e. transient objects), shadows, lightning changes, etc. In real-world scenarios, this assumption can hardly be satisfied. Even when carefully captured, recordings often contain moving people, cars, or other dynamic objects with their shadows, especially in locations that are tourist landmarks. Ignoring distractors during scene optimization results in undesired blurring effects and floating artifacts. At the same time, removing such distractors from the captured recordings is challenging and limits the widespread usage of NeRF and 3DGS. Manually annotating distractors is labor intensive. Another approach is to utilize pre-trained segmentation models to locate transient distractors. This approach has two main limitations: 1) it needs prior knowledge of transients as a semantic class, and 2), more importantly, existing segmentation models cannot distinguish between static and dynamic objects of the same semantic class. Additionally, we would like to identify semi-transient objects in recordings and remove them from the scene. We define a semi-transient object as an object that has both dynamic and static states during the capturing process, e.g. a pushed chair stops after some time and becomes a fully static object. Therefore, we need more robust identification methods for transient and semi-transient distractors throughout the captured recordings. We introduce T-3DGS, a novel approach for 3D static scene reconstruction from monocular video in uncontrolled settings. Our method includes an unsupervised transient detector and a transient mask propagation framework. Relying solely on image residuals for transient identification is unreliable due to issues such as appearance changes and color similarity to the background [ 20 , 22 ] . To address this issue, we develop a divergence-based technique on top of the uncertainty modeling approach [ 12 ] to detect transients. It helps improve mask accuracy and significantly reduce misclassifications of transient objects. Our experiments show that concurrent works [ 22 , 28 ] fail to remove semi-transient distractors (Fig. 1 ). We introduce a mask propagation framework for extracting object-aware masks that improve consistency in case of semi-transient distractors. Our method remains robust to all types of distractors. Additionally, we present the novel T-3DGS dataset with challenging scenes featuring semi-transient and slow-moving objects. Evaluations on both casual scenes [ 20 , 21 ] and our dataset show our method outperforms state-of-the-art approaches in reconstruction quality. Our key contributions, which together ensure consistent detection and removal of transient objects for improved 3D reconstruction, include: • Generalized uncertainty modeling for efficient transient object identification; • A divergence-based approach that leverages semantic consistency between reference and reconstructed frames for identifying transient objects; • A robust video object segmentation module that tracks objects across varying frame rates and semi-transient behaviors; • A challenging new dataset featuring diverse scenes with semi-transient distractors and slow-moving objects; • State-of-the-art performance on benchmark datasets for robust static scene reconstruction. 2 Related Work We provide a brief review of the works on Neural Radiance Fields and 3D Gaussian Splatting with a focus on removing non-static distractors in the scene. Neural Radiance Fields (NeRFs) [ 15 ] are widely adopted methods for high-quality scene reconstruction and novel view synthesis of 3D scenes. The seminal work 3D Gaussian Splatting [ 9 ] employs Gaussian primitives to model scenes instead of relying on continuous volumetric representations. This method has recently gained popularity as a faster alternative to NeRFs. Handling Distractors in NeRFs. NeRF-W [ 14 ] and RobustNeRF [ 21 ] are two pioneering works approaching the problem in a similar way. NeRF-W reconstructs both static background and transients combined with a data-dependent uncertainty field. RobustNeRF utilizes Iteratively Reweighted Least Squares for transient object identification and removal. Both methods rely on color residual supervision and frequently misclassify transient objects and backgrounds that share similar colors. Additionally, they both require careful hyper-parameters tuning. NeRF On-the-go [ 20 ] utilizes DINOv2 features [ 16 ] to identify and eliminate distractors by predicting uncertainties through a shallow MLP and can deal with more complex scenes than RobustNeRF. NeRF-HuGS [ 2 ] utilizes two types of heuristics: 1) COLMAP-based [ 23 ] features combined with SAM [ 11 ] and 2) residual-based heuristics to identify and remove transient distractors. Their method lacks robustness to heavy transient distractions, as both heuristics are unstable under such conditions, as demonstrated in [ 20 ] . Extracting Features from Vision Foundation Models . Vision Foundation Models (VFMs) are trained on large-scale data, enabling strong generalization to unseen domains or novel tasks. Task-agnostic models trained through self-distillation like DINO [ 1 , 16 ] learn features that can be generalized for multiple vision tasks. Video Object Segmentation . The goal of semi-supervised VOS is to identify when an object appears for the first time and then track it throughout the video. Several recent approaches based on transformers [ 3 , 4 ] have been proposed. However, current methods suffer from mask inconsistencies, particularly when objects disappear and reappear in the video. Additionally, these methods assume that the input has a high frame rate, and they become unstable when