DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting
Luis Wiedmann Luca Wiehe David RozenberszkiTechnical University of Munich {luis.wiedmann, luca.wiehe, david.rozenberszki}@tum.de · 2024
Read DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting by Luis Wiedmann Luca Wiehe David RozenberszkiTechnical University of Munich {luis.wiedmann, luca.wiehe, david.rozenberszki}@tum.de (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10972.txt.
Introduction
Luis Wiedmann, Luca Wiehe, and David Rozenberszki wrote this computer vision paper, published in two thousand twenty-four, on labeling objects in 3D scenes the system has never been taught. Robots and headsets need such open-set segmentation to work in new places.
Their pipeline, DCSEG, keeps the steps apart so each part can be swapped. First, the scene is rebuilt as 3D Gaussians, and features that ignore class are learned from a 2D proposal network. Those features are clustered into rough object masks. A 2D open-vocabulary model then names each cluster, so no retraining of the 3D part is needed. The split design yields multi-object masks for free and leans on the 3D shape for sturdiness. On indoor test sets it beat like NeRF-based systems, above all on rare classes.
Catalog record
| CREATOR | Luis Wiedmann Luca Wiehe David RozenberszkiTechnical University of Munich {luis.wiedmann, luca.wiehe, david.rozenberszki}@tum.de |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10972
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting Luis Wiedmann Luca Wiehe David RozenberszkiTechnical University of Munich {luis.wiedmann, luca.wiehe, david.rozenberszki}@tum.de Abstract Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. Recent advances introduce 3D Gaussian Splatting as a computationally efficient representation of the underlying scene. They enable the rendering of novel views while achieving real-time display rates and matching the quality of computationally far more expensive methods. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models. The pipeline proposes class-agnostic masks based on a 3D reconstruction of the scene. Given the resulting class-agnostic masks, we use a class-aware 2D foundation model to add class annotations to the 3D masks. We test this pipeline with 3D Gaussian Splatting and different 2D segmentation models and achieve better performance than more tailored approaches while also significantly increasing the modularity. Figure 1 : Decoupling the semantic segmentation pipeline. We present DCSEG, a holistic 3D reconstruction and scene understanding method. In the core of our method we leverage pretrained 2D foundation models such as SAM kirillov2023sam to recognize uniform semantic concepts in 2D images of 3D scenes, and use these predicted masks as contrastive optimization targets from multi-view images to class-agnostic 3D instances and object parts. These features are then used to cluster the Gaussians in 3D with hierarchical clustering methods. Simultaneously, we use a 2D semantic segmentation network to obtain class-aware masks and aggregate class-agnostic parts into meaningful semantic instances. As a results we obtain 2D/3D instance and semantic segmentation on synthetic and real world scenes. 0 0 footnotetext: * Equal contribution. Code available here . 1 Introduction Examples of common 3D representations include Point Clouds, Voxel Grids, Meshes, and Neural Radiance Fields (NeRFs) mildenhall2021nerf . NeRFs have become particularly prominent in the scientific community in recent years. This approach does not utilize explicit geometry to model the scene; instead, the scene is represented by an MLP. There have been several adaptations of NeRFs, but the basic principle remains the same. 3D Gaussian Splatting kerbl20233d is a fundamentally different approach compared to NeRFs. Instead of representing the scene implicitly in an MLP, Gaussian Splatting uses 3D Gaussians as an explicit representation. Given a set of images, we can initialize and optimize a set of 3D Gaussians for every scene. The key advantage is that this explicit representation doesn’t need volumetric rendering to project the 3D reconstruction onto 2D space. Instead, we can directly rasterize our explicit representation onto the image plane to render an image much faster. NeRFs and Gaussian Splatting have already been used as an underlying representation for scene understanding. NeRF-based approaches include LeRF, Panoptic NeRF, and OpenNeRF kerr2023lerf ; fu2022panoptic ; engelmann2024opennerf . OpenNeRF is built on the concepts of these latest methods. It speeds up standard pipelines by directly integrating the segmentation head onto the NeRF used to represent the scene. This integration enables rendering and segmentation in a single pass, and when combined with NeRF’s novel view synthesis capabilities, it achieves dominant performance. Recent research efforts, including ZegFormer ding2022zegformer and DeOP han2023deop , have focused on decoupling the mask proposal from the mask classification in the context of 2D image analysis. This decoupling is particularly advantageous in 3D space, where the diversity of 3D representations necessitates a more generalizable approach. The need for flexibility in 3D representations has led to a line of research that similarly separates mask proposal from mask classification, particularly in the realm of 3D instance segmentation, as demonstrated by works like OpenMask3D takmaz2023openmask3d and SAI3D yin2024sai3d . However, it is worth noting that these approaches do not utilize dense geometry but point clouds to perform segmentation, making them less adaptable for general use compared to 3D Gaussians. There are also some works that utilize 3D Gaussian Splatting as a basis for 3D semantic segmentation guo2024semanticgaussians ; qin2024langsplat ; li2024sgsslam . However, compared to our approach, they do not decouple mask proposal and mask classification, making them unadaptable to recent advances in mask classification architectures. Motivated by this, we add modularity to class-aware segmentation in 3D by decoupling the 3D semantic segmentation pipeline. Our contributions can be summarized as follows: • We utilize 3D Gaussian Splatting as an underlying representation for class-aware open-vocabulary semantic scene segmentation • We demonstrate that Gaussian Splatting can outperform comparable SOTA NeRF-based architectures for 3D semantic segmentation while being more modular • We present an architecture that can identify 3D instances and event parts without needing to train an instance-segmentation network 2 Related Work 3D Gaussian Splatting. Gaussian Splatting uses 3D Gaussians as an explicit geometry to represent a 3D scene. Given classical structure from motion (SfM) approaches, they take a sparse point cloud that is used to initialize a set of 3D Gaussians \mathcal{G}=\{\mathbf{g_{i}}\}_{i=1..k} where k is the total number of Gaussians. These 3D Gaussians can be efficiently rasterized to a given 2D image frame. This allows comparing reconstructed images to a set of ground truth images \mathcal{I} without requiring an expensive neural rendering approach as NeRFs do. An important feature of 3D Gaussian splatting is that the underlying rasterization approach is differentiable, enabling direct optimization of the 3D Gaussians based on the aforementioned comparison. This gradient-driven optimization is crucial for fine-tuning the Gaussians to reconstruct the scene accurately. Another advantage is the flexibility in the representation process: if a Gaussian is too large and causes a loss of detail, it can be split into smaller Gaussians for finer granularity. Conversely, if a single Gaussian is not enough to represent a structure, it can be duplicated to better capture the scene’s intricacies. Thus, the adoption of 3D Gaussians in our semantic segmentation pipeline is well-motivated by significant efficiency gains and flexibility. 3D Segmentation. Given a 3D reconstruction, semantic segmentation remains a challenging task (in particular, in an open-set fashion). Since there is way less training data available in 3D than in 2D, it is difficult to train an accurate segmentation network directly on 3D data. A common approach is integrating 2D foundation models into the training pipeline to leverage their more diverse and accurate training data set. SAGA uses this idea to segment 3D Gaussians. In essence, features are attached to 3D Gaussians for segmentation in 3D space. These features are then iteratively improved in a contrastive fashion utilizing a loss contrasting SAM masks kirillov2023sam and masks from 3D rendered onto the 2D image plane. Ground Truth OpenNeRF Ours (Openseg) Ours (OVSeg) Figure 2 : Segmentation results of our method compared to the ground truth and OpenNeRF. Our segmentation masks can detect boundaries more accurately e.g. the blanket/pillows or the wall behind the bed-lamps. Large uniform areas, such as the floor, can be detected with significantly less noise. Switching between Openseg and OVSeg can be done without retraining and demonstrates adaptability with respect to foundation models. Open Vocabulary Image Segmentation. Recent advances in Vision-Language Models (e.g., CLIP radford2021clip ) achieve impressive open vocabulary image-level classification performance. However, these models are unable to perform accurate pixel-wise segmentation. OpenSeg ghiasi2022scaling addresses these shortcomings by adding a layer that performs visual grouping of various concepts in an image before performing the final segmentation.