Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting

Runsong Zhu, Shi Qiu, Zhengzhe Liu, Ka-Hei Hui, Qianyi Wu, Pheng-Ann Heng, Chi-Wing Fu · 2025

Read Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting by Runsong Zhu, Shi Qiu, Zhengzhe Liu, Ka-Hei Hui, Qianyi Wu, Pheng-Ann Heng, Chi-Wing Fu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14029.txt.

Introduction

Runsong Zhu and co-authors lift flat image segments into 3D scenes in a 2025 paper. Gaussian splatting builds scenes from soft 3D blobs that render fast and sharp. Sorting those blobs by object, straight from 2D masks, had relied on brittle matching or heavy two-stage pipelines.

Their Unified-Lift learns a feature for each blob that encodes which object it belongs to. A learnable codebook holds one entry per object, giving the scene an explicit object-level read. Two extra modules clean up codebook learning and filter noisy labels. On three benchmarks the method beat rivals in both quality and speed. Code is public.

Catalog record

CREATORRunsong Zhu, Shi Qiu, Zhengzhe Liu, Ka-Hei Hui, Qianyi Wu, Pheng-Ann Heng, Chi-Wing Fu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting Runsong Zhu, Shi Qiu, Zhengzhe Liu, Ka-Hei Hui, Qianyi Wu, Pheng-Ann Heng, Chi-Wing Fu Abstract Lifting multi-view 2D instance segmentation to a radiance field has proven to be effective to enhance 3D understanding. Existing methods rely on direct matching for end-to-end lifting, yielding inferior results; or employ a two-stage solution constrained by complex pre- or post-processing. In this work, we design a new end-to-end object-aware lifting approach, named Unified-Lift that provides accurate 3D segmentation based on the 3D Gaussian representation. To start, we augment each Gaussian point with an additional Gaussian-level feature learned using a contrastive loss to encode instance information. Importantly, we introduce a learnable object-level codebook to account for individual objects in the scene for an explicit object-level understanding and associate the encoded object-level features with the Gaussian-level point features for segmentation predictions. While promising, achieving effective codebook learning is non-trivial and a naive solution leads to degraded performance. Therefore, we formulate the association learning module and the noisy label filtering module for effective and robust codebook learning. We conduct experiments on three benchmarks: LERF-Masked, Replica, and Messy Rooms datasets. Both qualitative and quantitative results manifest that our Unified-Lift clearly outperforms existing methods in terms of segmentation quality and time efficiency. The code is publicly available at https://github.com/Runsong123/Unified-Lift . 1 Introduction Accurate 3D scene segmentation enhances scene understanding and facilitates scene editing, benefiting many downstream applications in virtual reality, augmented reality, and robotics. However, accurate 3D scene segmentation is challenging to obtain, due to limited 3D dataset size and labor-intensive manual labeling in 3D. To bypass these challenges, recent studies [ 44 , 33 , 1 ] suggest lifting 2D segmentations predicted by foundation models [ 15 , 3 ] to the 3D scene modeled by a radiance field for instance-level understanding. Yet, 2D instance segmentations predicted by models like SAM [ 15 ] lack consistency across different views, e.g . , the same object may have different IDs when viewed from different angles, leading to conflicting supervision. Besides, inferior segmentations, e.g . , under- or over-segmentation, make the lifting process challenging. Various strategies have been proposed to address the above issues. An early work Panoptic Lifting [ 33 ] trains a NeRF to render instance predictions in an end-to-end manner and matches the model’s 3D predictions with the initial 2D segmentation masks to provide supervision. However, this approach tends to produce multi-view inconsistent segmentation as it is highly sensitive to the variances of matching results. Subsequently, [ 39 , 23 ] propose object association techniques as a preprocessing to prepare view-consistent 2D segmentation maps with improved multi-view consistency (see Fig. 1 (b)). However, the preprocessing stage often struggles to produce accurate results and the accumulated error can further degrade the performance. The recent state-of-the-art methods [ 1 , 40 ] encode instance information in the feature field using contrastive learning and apply a clustering as a postprocessing to produce the final segmentations (see Fig. 1 (c)). Figure 1 : Comparing the pipeline of our method against previous lifting solutions. Though significant improvements are achieved, their performance is always constrained by the naive clustering postprocess, which is hyperparameter-sensitive and also induces error accumulation. Given the above concerns, we come up with this question: “Can we have an end-to-end lifting framework for accurate 3D scene segmentation, without the need of pre- or post-processing?” In this work, we propose a new end-to-end object-aware lifting method called Unified-Lift for accurate 3D scene segmentation, facilitating the generation of coherent and view-consistent instance segmentation across different views. We exploit the recent advancement of the radiance field, i.e . , 3D Gaussian splatting (3D-GS) [ 12 ] , as the 3D scene representation due to its superior efficiency and rendering quality. Specifically, we augment each Gaussian point in 3D-GS with a Gaussian-level feature and learn these features using contrastive learning defined in each individual view. In particular, we introduce a novel global object-level codebook to represent each object in the 3D scene. This codebook is further associated with the rendered Gaussian-level features to predict segmentation results, enhancing object-level awareness during training. However, effective codebook learning is non-trivial. Naive training solutions can lead to suboptimal performance. Hence, we present effective learning strategies to optimize the object-level codebook. First, we introduce a novel association learning module, in which we design an area-aware ID mapping algorithm to generate pseudo-labels of association with enhanced multi-view consistency. Additionally, we present two complementary loss functions, i.e . , sparsity and concentration parts, to achieve more reliable object-level understanding. Second, we design a novel noisy label filtering module to enhance the robustness of our method by estimating an uncertainty map for the segmentation masks, leveraging the learned Gaussian-level features in a self-supervised manner. During inference, we obtain novel-view instance segmentation results without any pre- or post-processing, effectively avoiding error accumulation. To evaluate the effectiveness of our Unified-Lift, we conduct experiments on the widely-used LERF-Masked [ 39 ] dataset and the indoor scene dataset, Replica [ 35 ] . Both quantitative and qualitative results demonstrate that our Unified-Lift outperforms all the existing lifting methods by a notable margin. Furthermore, we conduct additional experiments on the challenging Messy Rooms dataset [ 1 ] , where each scene contains up to 500 objects, demonstrating the scalability of Unified-Lift in handling large numbers of objects. Our main contributions are summarized as follows: • We propose a new end-to-end object-aware lifting method (named Unified-Lift) for accurate 3D scene segmentation by jointly learning the Gaussian-level features and a global object-level codebook. • We present a novel association learning module and a noisy label filtering module to facilitate effective learning of the object-level codebook. • We set a new state-of-the-art performance on multiple datasets and demonstrate strong scalability in handling large numbers of objects without the need of pre- or post-processing. 2 Related Works Radiance field: from implicit to explicit. Radiance field emerges as a promising representation for reconstructing 3D scenes with various properties, e.g . , geometries, colors, and semantics, from only 2D inputs such as RGB images and segmentation masks. Neural Radiance Field (NeRF) [ 25 ] models the radiance field using a neural network composed of layers of multilayer perceptrons. Since then, various works attempt to improve the efficiency of NeRF, e.g . , by explicitly formulating the field using 3D structures such as voxels [ 2 , 22 ] and hash grids [ 27 ] . Later on, 3D Gaussian Splatting (3D-GS) [ 12 , 38 , 21 , 43 , 5 , 11 , 41 ] is introduced to model the radiance field as a set of explicit Gaussian points. This approach allows for a splatting-style rendering [ 16 ] , which is highly efficient and demonstrates great potential of real-time rendering. Given the advantages, we employ 3D-GS as the backbone representation in our framework for creating consistent 3D segmentations. Segmentation: from 2D to 3D. Segmentation is a long-standing task in computer vision research. Recent progress witnesses advancements in 2D, thanks to the availability of large-scale datasets. Notably, various foundation models, such as SAM [ 15 ] and its subsequent works [ 37 , 18 ] , show great performance in numerous 2D

Read the complete text