Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors
Jameel Hassan, Yasiru Ranasinghe, Vishal Patel · 2026
Read Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors by Jameel Hassan, Yasiru Ranasinghe, Vishal Patel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30638.txt.
Introduction
Jameel Hassan, Yasiru Ranasinghe and Vishal Patel give 3D scenes a language interface without heavy feature training. The scenes are built with 3D Gaussian Splatting, which represents a scene as clouds of soft 3D blobs that render fast. Prior language extensions distilled CLIP features, bulky image-text embeddings, into every blob. That limited queries to simple noun phrases and made instance grouping noisy or preset.
GaussDet instead leans on 2D detectors that already answer open-vocabulary and referring queries, such as the red chair by the door. It learns features that split Gaussians into 3D instance groups. Each group is rendered from many views, and 2D detections vote on its label, forming a view-aggregated label distribution per instance. This voting smooths out bad labels from imperfect grouping. In strict zero-shot tests, referring-expression grounding improved by a wide margin over prior methods.
Catalog record
| CREATOR | Jameel Hassan, Yasiru Ranasinghe, Vishal Patel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30638
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors Jameel Hassan, Yasiru Ranasinghe, Vishal Patel Abstract 3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet , a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks—open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF)—demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting. 1 Introduction Open-vocabulary capabilities have garnered significant attention across numerous computer vision domains, including image [ 42 , 41 , 19 ] and video understanding [ 6 , 23 , 33 ] , as well as 3D scene understanding [ 31 , 30 , 36 , 5 , 11 ] . Within the domain of neural rendering, the fast training, explicit point-based representation, and real-time rendering capabilities of 3D Gaussian Splatting (3DGS) [ 16 ] have driven its rapid adoption for 3D/4D reconstruction [ 15 , 22 , 26 ] , generation [ 39 , 24 ] , and scene understanding [ 21 , 43 ] . Consequently, enabling open-vocabulary comprehension within 3DGS serves as a critical stepping stone for real-world applications such as embodied AI [ 35 , 12 ] and autonomous driving [ 14 , 45 ] . Early efforts to enhance 3DGS with language-grounding capabilities focused on incorporating a learnable language attribute to individual Gaussians [ 32 , 44 , 38 , 34 ] . However, these strategies remained largely 2D-centric, relying on rendering language features onto 2D image planes for query evaluation. Pivoting toward a more 3D-native approach, OpenGaussian [ 37 ] disentangled instance and semantic properties by first learning an instance attribute supervised via view-agnostic SAM masks [ 17 ] . These instance features, combined with 3D coordinates, are clustered using k -means to group Gaussians into instances, with each instance group 1 1 1 The term “instance group” is used to refer to all Gaussians belonging to a single instance. assigned a 2D CLIP feature via a rule-based view selection strategy. Building on this, LaGa [ 8 ] adopts a similar instance-learning paradigm but eliminates the need to predefine the cluster count by grouping Gaussians through a 2D-3D association, clustering rendered SAM feature masks using HDBSCAN [ 27 ] . Despite these differences, both methods fundamentally rely on dense CLIP features to assign semantics to 3D instances. This dependence introduces severe bottlenecks, limiting open-vocabulary querying by (1) restricting queries strictly to simple noun phrases [ 40 ] , and (2) suffering from low semantic expressiveness due to the loss of surrounding image context when extracting CLIP features from isolated SAM masks. Motivated by recent breakthroughs in open-vocabulary object detection [ 10 , 29 , 2 ] , we propose GaussDet . Our method leverages the 2D-3D associations between Gaussian instance groups and 2D bounding box detections to perform robust open-vocabulary segmentation and referential expression grounding in 3DGS. Specifically, we utilize an open-vocabulary object detector to generate discrete semantic label maps across all image frames, and follow the grouping mechanism of [ 8 ] to obtain 3D Gaussian instance groups. We render these instance groups onto the 2D image planes and aggregate spatial votes from the discrete label maps to construct a View-Aggregated Semantic Label Distribution (VASD). The proposed VASD serves as a robust method to counteract the spurious semantic label votes caused by low-quality instance groupings, improving the overall semantic label quality. We present a sample visualization of our method in Fig. 1 . Our primary contributions are as follows: • We introduce a novel, 2D object detector-based approach for the semantic labeling of 3D Gaussian splats, significantly improving open-vocabulary segmentation without relying on dense CLIP features. • We demonstrate the seamless extension of our pipeline to referential expression grounding in 3DGS within a zero-shot setting. • Extensive evaluations show that GaussDet achieves substantial performance gains over existing 3DGS open-vocabulary methods, including a notable absolute increase of 16.7\% mIoU on the Ref-LeRF benchmark compared to the recent, per-scene optimized ReferSplat [ 13 ] . Figure 1 : Open-vocabulary and referential expression grounding with GaussDet . Referential grounding examples for “ Toy cat statue ” and “ Waldo ” are shown. 2 Related Work Neural Rendering: The introduction of Neural Radiance Fields (NeRF) [ 28 ] marked a paradigm shift in novel view synthesis and 3D scene representation, achieving highly photorealistic renderings. While numerous subsequent works sought to improve rendering quality of NeRFs [ 3 , 4 ] , the implicit continuous representation of NeRFs inherently relies on dense ray marching, leaving training and rendering computationally expensive. The recent developments in 3D Gaussian Splatting (3DGS) [ 16 ] represents scenes explicitly using a set of anisotropic 3D Gaussians, optimized via a differentiable tile-based rasterizer. This explicit formulation achieves state-of-the-art rendering quality and real-time frame rates and provides a structured, point-like representation that is highly amenable to downstream 3D scene understanding and object-level manipulation. Consequently, 3DGS has been rapidly adopted across a wide range of real-world applications, including autonomous driving [ 14 , 45 ] and embodied AI [ 35 , 12 ] . Open-Vocabulary Segmentation of Gaussian Splats: Initial efforts to equip 3DGS with open-vocabulary understanding adopted 2D-centric strategies. Methods like LangSplat [ 32 ] and Feature 3DGS [ 44 ] append high-dimensional semantic features directly to individual Gaussians for 2D query evaluation. Recent works transition to a 3D-centric approach, disentangling instance learning from semantic assignment. OpenGaussian [ 37 ] groups Gaussians using k -means clustering on SAM-supervised instance features, but requires cluster count ( K ). A bottom-up instance aggregation is proposed in [ 20 ] to overcome this. Meanwhile, [ 8 ] bypasses k by clustering rendered SAM mask features via HDBSCAN; however, this 2D-3D association frequently leads to under-segmentation and captures spurious masks. Crucially, these 3D-centric methods remain bottlenecked by their reliance on dense CLIP features for the final semantic assignment, which restricts queries to simple noun phrases. Referring Expression Segmentation: Referring segmentation grounds complex, descriptive natural language queries into precise object masks. While highly successful in 2D using vision-language fusion and large multimodal models [ 25 , 18 ]