SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians
Siyun Liang, Sen Wang, Kunyi Li, Michael Niemeyer, Stefano Gasperini, Nassir Navab, Federico Tombari · 2024
Read SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians by Siyun Liang, Sen Wang, Kunyi Li, Michael Niemeyer, Stefano Gasperini, Nassir Navab, Federico Tombari (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10231.txt.
Introduction
Liang and co-authors teach computers to cut 3D scenes into named parts using plain language in this 2024 paper. A popular rendering method builds scenes from millions of soft blobs, but attaching word meanings to each blob eats memory. Their system, SuperGSeg, separates the jobs.
First, neural blobs learn shape, object identity, and grouping from photos plus ready-made masks. A sparse set of super-blobs then carries language features lifted from flat images into the scene. This keeps memory moderate while supporting layered understanding. Tests showed strong results on picking objects by description and on labeling scene regions. The design suits interactive tools that must answer open-ended language queries about 3D spaces.
Catalog record
| CREATOR | Siyun Liang, Sen Wang, Kunyi Li, Michael Niemeyer, Stefano Gasperini, Nassir Navab, Federico Tombari |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10231
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians Siyun Liang, Sen Wang, Kunyi Li, Michael Niemeyer, Stefano Gasperini, Nassir Navab, Federico Tombari Abstract 3D Gaussian Splatting has recently gained traction for its efficient training and real-time rendering. While the vanilla Gaussian Splatting representation is mainly designed for view synthesis, more recent works investigated how to extend it with scene understanding and language features. However, existing methods lack a detailed comprehension of scenes, limiting their ability to segment and interpret complex structures. To this end, we introduce SuperGSeg, a novel approach that fosters cohesive, context-aware scene representation by disentangling segmentation and language field distillation. SuperGSeg first employs neural Gaussians to learn instance and hierarchical segmentation features from multi-view images with the aid of off-the-shelf 2D masks. These features are then leveraged to create a sparse set of what we call Super-Gaussians. Super-Gaussians facilitate the distillation of 2D language features into 3D space. Through Super-Gaussians, our method enables high-dimensional language feature rendering without extreme increases in GPU memory. Extensive experiments demonstrate that SuperGSeg outperforms prior works on both open-vocabulary object localization and semantic segmentation tasks. Please visit our project for more results. Figure 1 : We present SuperGSeg , a novel method that clusters similar Gaussians into superpoint-like representations, termed Super-Gaussians. By leveraging Super-Gaussians, our approach effectively integrates diverse feature fields to achieve a comprehensive understanding of 3D scenes. SuperGSeg supports a wide range of functionalities, including open-vocabulary semantic segmentation, both promptable and promptless instance segmentation, and finer-grained hierarchical segmentation. 1 Introduction 3D Gaussian Splatting (3DGS) kerbl20233d has rapidly gained recognition in novel view synthesis over NeRF mildenhall2020nerf for its efficient training, real-time rendering, and explicit point-based representation. These attributes make 3DGS well-suited for a broad range of applications, such as 3D reconstruction yu2024gaussian ; dai2024high ; guedon2024sugar , content generation yi2024gaussiandreamerpro , and scene understanding qin2024langsplat ; shi2024language ; zhou2024feature ; gaussian_grouping ; wu2024opengaussian . A particularly promising direction involves extending 3DGS frameworks to open-vocabulary understanding, empowering the interaction with the rendered scenes through flexible, language-based queries radford2021clip ; li2022languagedriven . Significant efforts have been made to distill language features into 3DGS from both 2D zuo2024fmgs ; qin2024langsplat ; li2024geogaussian ; zhou2024feature and 3D wu2024opengaussian perspectives. 2D methods achieve this by transferring 2D language features to 3D through the multi-view consistency of 3DGS rendering, which is based on an explicit 3D point representation. To achieve this, they typically reduce the language feature maps from their original high dimensionality (e.g., 512D) to a lower dimension (e.g., 3D) qin2024langsplat using an autoencoder or a convolutional network zhou2024feature . However, this dimensionality reduction can result in losing important information about language features. Another limitation of this approach is its inability to recognize occluded objects. OpenGaussian wu2024opengaussian addresses this issue by focusing on 3D point-level open-vocabulary tasks. By introducing a coarse-to-fine codebook optimized using masks generated by SAM kirillov2023sam , OpenGaussian assigns language features to instance-level Gaussian Splatting, enabling direct language queries on the 3D point cloud. However, two problems remain in OpenGaussian: 1) Only sparse language features can be queried at the 3D instance level, making pixel-level semantic segmentation tasks impractical. 2) Its decoupling of the alpha blending for language queries introduces challenges due to ambiguities in the language features assignment in regions with overlapping Gaussians, resulting in unreliable semantic predictions. In this paper, we present a superpoint-guided language distillation method to improve fine-grained scene understanding in 3D environments. Building on LangSplat qin2024langsplat , we begin by generating pixel-space masks using an off-the-shelf segmentation model but move beyond fixed granularities, incorporating multi-level masks for enhanced detail capture in both instance and part segmentation. Our approach assigns each Gaussian instance and part features through contrastive learning bhalgat2023contrastive , translating 2D segmentation insights into 3D. Inspired by Super points, we cluster Gaussians into Super-Gaussians based on spatial and instance characteristics, allowing for robust scene representation. By assigning high-dimensional language features to each Super-Gaussian, we support detailed, comprehensive scene understanding without loss of language information, enhancing structural integrity in complex scenes. The main contributions of this paper are summarized as follows: • We propose SuperGSeg: a 3D segmentation method with neural Gaussians, designed to learn hierarchical instance segmentation features from 2D foundation models. • We introduce the concept of Super-Gaussian, a novel representation that integrates hierarchical instance segmentation features, enabling the embedding of high-dimensional language features. This approach addresses previously unfeasible challenges in representing complex scenes with rich semantic details. • Extensive experiments on the LERF-OVS and ScanNet datasets demonstrate the effectiveness of the proposed method, achieving significant improvements in open-vocabulary 3D object-level and scene-level semantic segmentation, with particular strength in capturing fine-grained scene details and dense pixel semantic segmentation tasks for the first time. 2 Related Work 3D Gaussian Splatting Neural Radiance Fields (NeRF) mildenhall2020nerf combines neural implicit representations with differentiable volumetric rendering. In recent, 3D Gaussian Splatting (3DGS) kerbl20233d introduced a new explicit representation of 3D scenes via a set of 3D Gaussians. By optimizing their position and appearance, it leads to higher-quality renderings in real-time. Building on 3DGS, many recent works have focused on either enhancing the rendering quality yu2024mip ; huang20242d ; li2024geogaussian or the reconstruction performance yu2024gaussian ; dai2024high ; guedon2024sugar . While these approaches offer advantages in small-scale scenes, they tend to excessively expand Gaussian blobs to fit all training views, leading to redundancy and limiting scalability in complex, large-scale scenes. To overcome this, Scaffold-GS lu2024scaffold introduced a hierarchical, region-aware 3D scene representation using anchor points, each linked to neural Gaussians with learnable offsets and attributes (opacity, color, rotation, scale) based on anchor features and view. The proposed SuperGSeg builds upon Scaffold-GS representation and brings it to a new level by embedding semantic and language features. Unlike prior works, this enables the query of the 3D scene via language and performs 3D semantic segmentation. Open-Set Segmentation LERF kerr2023lerf and related methods goel2023isrf ; kobayashi2022dff ; takmaz2023openmask3d embed language features radford2021clip into 3D space to enable interactive open-vocabulary segmentation. However, since these language features are trained on full images without explicit boundary guidance, they often yield noisy segmentation boundaries and lack the ability to distinguish between instances due to missing instance-level supervision. SPInNeRF mirzaei2023spinnerf addresses this by initializing 2D masks via video segmentation and lifting them into 3D with a NeRF mildenhall2020nerf , followed by multi-view refinement to achieve consistent 3D segmentation. Similarly, SA3D cen2023samin3d introduces an online interactive segmentation method that propagates a SAM kirillov2023sam mask into 3D and across other views iteratively. However, these approaches are sensitive to the choice of reference view and struggle with complex cases involving severe occlusions. OmniSeg3D ying2024omniseg3d is a universal segmentation method that transforms inconsistent 2D segmentations into a globally consistent 3D feature field. It employs a hierarchical representation and contrastive learning framework, enabling robust hierarchical 3D understanding and high-quality object segmentation. However, OmniSeg3D builds upon NeRF, which requires longer training and does not integrate language features. Figure 2 : SuperGSeg Overview. We initialize the 3D Gaussians from a sparse set of anchor points, each generating k Gaussians with corresponding attributes. First, we train the appearance and segmentation features using RGB images and segmentation masks generated by SAM kirillov2023sam . Next, we use the segmentation features and their spatial positions to produce a sparse set