Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Querying
Hairong Yin Huangying Zhan Yi Xu Raymond A. Yeh · 2025
Read Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Querying by Hairong Yin Huangying Zhan Yi Xu Raymond A. Yeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21767.txt.
Introduction
A robot that hears "hand me the red mug" must find that object inside its three-dimensional map of the room. Hairong Yin, Huangying Zhan, Yi Xu, and Raymond Yeh improved that skill in 2025 for scenes built from Gaussian splats, a fast rendering format. Labels painted onto the scene from camera views often disagree with each other, blurring the map's meanings.
Their method tracks segmentation masks across views to build one consistent set of labels for teaching the scene. It then answers queries by first finding that trusted ground truth and using it to pick the matching points. Scores on three benchmark datasets beat prior methods, with one dataset gaining over twenty points.
Catalog record
| CREATOR | Hairong Yin Huangying Zhan Yi Xu Raymond A. Yeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Querying Hairong Yin Huangying Zhan Yi Xu Raymond A. Yeh Abstract Open-vocabulary querying in 3D Gaussian Splatting aims to identify semantically relevant regions within a 3D Gaussian representation based on a given text query. Prior work, such as LangSplat, addressed this task by retrieving these regions in the form of segmentation masks on 2D renderings. More recently, OpenGaussian introduced point-level querying, which directly selects a subset of 3D Gaussians. In this work, we propose a point-level querying method that builds upon LangSplat’s framework. Our approach improves the framework in two key ways: (a) we leverage masklets from the Segment Anything Model 2 (SAM2) to establish semantic consistent ground-truth for distilling the language Gaussians; (b) we introduces a novel two-step querying approach that first retrieves the distilled ground-truth and subsequently uses the ground-truth to query the individual Gaussians. Experimental evaluations on three benchmark datasets demonstrate that the proposed method achieves better performance compared to state-of-the-art approaches. For instance, our method achieves an mIoU improvement of + 20.42 +20.42 on the 3D-OVS dataset. Project page see: h t t p s : / / e v e l i n y i n . g i t h u b . i o / s e c o n G S / \hrefhttps://evelinyin.github.io/seconGS/ † † Part of this work was done at OPPO US Research Center. 1 Introduction Open-vocabulary 3D scene understanding is the task of interpreting a 3D scene by associating its components with natural language. A system that understands 3D scenes leads to many potential applications in robotics, graphics editing, human-computer interaction, etc. This task requires accurately interpreting complex 3D geometries and aligning them with language descriptions. Recently, NeRF [ 43 , 26 , 25 , 41 ] and 3D Gaussians [ 15 , 46 , 4 ] have emerged as popular 3D scene representations due to their high visual qualities and fast rendering speeds. This led to considerable interest [ 23 , 35 , 28 , 6 , 52 , 40 ] in building open-vocabulary understanding capabilities on top of these 3D scene representations. With advancements in foundation 2D vision-language models (VLMs) [ 21 , 29 ] and 3D-aware models [ 16 ] , recent works [ 28 , 35 , 51 ] have further explored this area by incorporating language understanding into 3D representations through the introduction of language parameters. To evaluate open-vocabulary capabilities, these works have formulated the task of open-vocabulary querying. Figure 1 : Visualization of the language embedding supervision. For the “red bag” circled in yellow, the ground-truth constructed by LangSplat [ 28 ] is inconsistent across frames while Ours remains consistent across frames. Given a text query and a language-augmented 3D scene representation, LangSplat [ 28 ] formulates the task as retrieving the pixels related to the query, represented as a 2D mask. In this case, language features are rendered into an image, and the querying procedure involves comparing the query vector with the language feature at each pixel to identify relevant regions. However, 2D querying is limited to capturing 3D spatial information accurately, making it inadequate for tasks requiring precise 3D understanding. This leads to the focus on direct querying in 3D [ 44 , 40 , 22 ] , i.e . , point-level quering, where they identify relevant 3D parameters directly related to a text query. For 3D Gaussian Splatting, the 3D querying task involves retrieving the relevant Gaussians. The primary advantage of direct querying in 3D is that associations are made directly to the 3D representation rather than through a rendered image. This could be beneficial for tasks that require direct access to 3D data, e.g . , robotics tasks [ 34 , 12 , 38 ] , object editing [ 39 , 19 , 42 ] , object selection [ 31 ] , etc. In this work, we propose a method, based on LangSplat’s framework, that leverages the power of Segment Anything Models [ 18 , 30 ] to construct consistent language embeddings across different views. This consistency ensures robust supervision for training the language Gaussians, where the same embedding would be used to supervise the same object; See Fig. 1 for comparison with LangSplat. Furthermore, we introduce a two-step querying process to mitigate the challenge of selecting an effective similarity threshold between the text query embedding and the learned language embedding in LangSplat. Empirically, we evaluate our method on three datasets: LERF [ 16 ] , 3D-OVS [ 23 ] , and Replica [ 37 ] , demonstrating that it outperforms the state-of-the-art method in terms of mIoU by +4.16,+20.42 , and +1.74 , respectively. Our main contributions are as follows: • We introduce a method for generating semantic and 3D-consistent ground-truth to train language-aware Gaussians. • With this improved 3D language Gaussians, we propose an effective two-step querying process by leveraging the created consistent ground-truth. • Extensive experiments across multiple datasets demonstrate that our approach outperforms existing open-vocabulary 3D querying methods. 2 Related Work Vision foundation models. In computer vision, several open-sourced foundation models have become the bedrock of many works. These foundation models’ capabilities can either be used directly or their features can be distilled into another model. In language and vision, CLIP [ 29 ] is a model that is capable of encoding images and natural language text to the same embedding space. Using this joint embedding space, they demonstrate zero-shot capability for image classification, which later is generalized to segmentation [ 50 ] and language segmentation (LSeg) [ 21 ] . In the area of image segmentation, the Segment Anything Model (SAM) [ 18 ] is a notable foundation model. SAM’s segmentation capabilities have been adapted and extended to 3D tasks. Recent works [ 3 , 17 ] have leveraged SAM to integrate semantic information into NeRFs, enabling the extraction of 3D masks for target objects. Other approaches [ 24 , 11 ] have incorporated semantic features into point clouds, enhancing object representation and segmentation in 3D. The process in 3D Gaussian Splatting [ 15 ] has further motivated studies [ 9 , 45 , 2 , 14 , 47 , 33 ] focus on object representation in both 3D and 4D, including advancements in interactive segmentation and object tracking. Recently, SAM2 [ 30 ] extended SAM’s capability to tracking of masklets, i.e . , consistent masks across both space and time. In this work, we leverage masklets extracted by SAM2 to provide consistent guidance for training the language Gaussians. Open-vocabulary 3D scene understanding. With advancements in 3D scene representation, there is a surge in interest in incorporating semantics/language into 3D representation. One line of work [ 16 , 23 , 34 , 43 , 6 ] distilled features from DINO [ 1 ] , LatentDiffusion [ 32 ] , LSeg [ 21 ] , CLIP [ 29 ] to learn a NeRF [ 26 ] , or leveraged 2D annotations [ 49 , 36 ] to construct feature/language fields. Others [ 27 , 48 , 13 , 7 ] distills knowledge from these language-rich models into point clouds, enabling open-vocabulary 3D scene understanding. In more recent works [ 28 , 35 , 51 , 10 ,