Zero-Shot VISUAL GROUNDING in 3D Gaussians via View Retrieval

Liwei Liao, Xufeng Li, Xiaoyun Zheng, Boning Liu, Feng Gao \sthanks Corresponding Author, Ronggang Wang · 2025

Read Zero-Shot VISUAL GROUNDING in 3D Gaussians via View Retrieval by Liwei Liao, Xufeng Li, Xiaoyun Zheng, Boning Liu, Feng Gao \sthanks Corresponding Author, Ronggang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15871.txt.

Introduction

Liwei Liao leads a team that finds objects in 3D scenes from text prompts, with no task training. A scene built as 3D Gaussians stores its look in a spread out, implicit way. Past finders needed training for each scene and large labeled sets. Their frame turns the 3D search into a 2D retrieval problem across views.

For each view of the scene, they retrieve object level clues that match the prompt. Pooling clues across views pins the object in 3D. No per scene training and no 3D annotation are needed. Tests reach top scores among rivals while skipping that training. The work lays a base for zero shot 3D grounding. Video demos are shared online.

Catalog record

CREATORLiwei Liao, Xufeng Li, Xiaoyun Zheng, Boning Liu, Feng Gao \sthanks Corresponding Author, Ronggang Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot VISUAL GROUNDING in 3D Gaussians via View Retrieval Liwei Liao, Xufeng Li, Xiaoyun Zheng, Boning Liu, Feng Gao \sthanks Corresponding Author, Ronggang Wang Abstract 3D Visual Grounding (3DVG) aims to locate objects in 3D scenes based on text prompts, which is essential for applications such as robotics. However, existing 3DVG methods encounter two main challenges: first, they struggle to handle the implicit representation of spatial textures in 3D Gaussian Splatting (3DGS), making per-scene training indispensable; second, they typically require larges amounts of labeled data for effective training. To this end, we propose G rounding via V iew R etrieval (GVR), a novel zero-shot visual grounding framework for 3DGS to transform 3DVG as a 2D retrieval task that leverages object-level view retrieval to collect grounding clues from multiple views, which not only avoids the costly process of 3D annotation, but also eliminates the need for per-scene training. Extensive experiments demonstrate that our method achieves state-of-the-art visual grounding performance while avoiding per-scene training, providing a solid foundation for zero-shot 3DVG research. Video demos can be found in https://github.com/leviome/GVR_demos . † † address: Author Affiliation(s) † † address: 1 Peking University, 2 Peng Cheng Laboratory, 3 City University of Hongkong 1 Introduction 3D Gaussian Splatting (3DGS) [ 1 ] offers significant advantages over traditional 3D reconstruction methods in terms of reconstruction speed, quality, and rendering efficiency. However, because 3DGS adopts a semi-implicit representation with explicit geometric structures and implicit spatial textures, it is difficult to directly apply previous point-cloud or radiance-field methods to 3DGS. In recent years, there has been a growing body of work on developing scene understanding algorithms [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] specifically for 3DGS. These approaches can be categorized into two main types: (1) supervised 3DVG methods such as SceneSplat [ 10 ] that require annotated 3D data for training and perform end-to-end inference, and (2) per-scene training 3DVG methods such as LangSplat [ 11 ] that train a language Gaussian field for each specific scene. However, both the two approaches face different limitations: (1) For SceneSplat, pre-training is conducted only on 7,000 indoor scenes, so its generalization ability is limited to several indoor environments. To expand its generalization requires a massive amount of 3D annotated data, which is currently expensive. (2) For LangSplat, hour-scale per-scene preparation (including preprocessing and training) are required, and the query stage is also computationally complex. These two limitations severely restrict the application of 3DVG in the 3DGS domain, especially the requirement for hours of per-scene training, which greatly undermines its practicality. To address the two limitations, we propose a retrieval-based 3D Gaussian visual grounding method that reformulates the 3DVG task as a 2D retrieval problem, enabling effective visual grounding in 3D Gaussian scenes. For the first limitation, our work is training-free by leveraging existing mature 2D visual foundation models to accomplish visual perception, thereby largely preserving the generalization ability of these 2D models. For the second limitation, we replace per-scene training with a knowledge book approach, which not only greatly reduces preparation time for each scene but also accelerates the query process. Our main contributions are summarized as follows: • We introduce GVR , to the best of our knowledge, the first framework capable of zero-shot visual grounding in 3D Gaussian scenes. • We propose a view retrieval mechanism for 3DVG, which eliminates the need for 3D annotated data and enables high-quality visual grounding using only existing mature 2D visual foundation models. • Extensive experiments demonstrate that our approach achieves state-of-the-art performance in zero-shot 3D visual grounding of 3DGS, while significantly saving training time. 2 Methodology Our method aims to address the problem of zero-shot 3D visual grounding specifically for 3DGS. The overall process can be formulated as: \textbf{I}_{tar}=\operatorname{GVR}(Q;\textbf{G}) (1) where \textbf{I}_{tar}\in\mathbb{R}^{N\times 1} indicates the flag of all Gaussians that 1 denotes target and 0 denotes background, Q is the text query (e.g., a red apple) describing the desired object or region, \textbf{G}\in\mathbb{R}^{N\times 59} denotes the 3D Gaussian scene containing N Gaussians, and \text{GVR}(\cdot) refers to our proposed 3DVG method. Figure 1: Overall pipeline of our proposed GVR . (A) Preparation : Construct a Semantic Vector Book using SAM and CLIP, and obtain a Depth Book via 3DGS depth rendering. (B) Query : Encode the text query with CLIP’s text encoder, compute similarity with the multi-view semantic vector library, select top-k relevant views and object patches, and estimate the 3D object location via multi-view back-projection and stereo voting. (C) Segmentation : Perform point-driven segmentation on the BEV using the projected 3D location, obtain coarse target Gaussians via frustum filtering, and refine them through surrounding multi-view frustum intersection. 2.1 (A) Preparation: Knowledge Books Building As shown in Fig. 1 (A), we first prepare Semantic Vector Book (SVB) and Depth Book (DB) \{\textbf{B}_{s},\textbf{B}_{d}\} . Since the 3D scene G is reconstructed from multi-view images and camera parameters \{\mathbf{V},\mathbf{C}\} , all semantic information about the scene can be inferred from these multi-view images. Based on this observation, we construct a SVB \textbf{B}_{s}\in\mathbb{R}^{n\times m\times c} to store the scene’s semantic information. Specifically, we use SAM [ 12 ] to segment each view and obtain multiple object masks, then encode each object patch into a semantic vector using CLIP’s [ 13 ] image encoder, as: \textbf{B}^{(i)(j)}_{s}=\mathcal{E}_{img}(\operatorname{SAM}(v_{i})^{(j)}) (2) where \operatorname{SAM}(\cdot) denotes the SAM model that segments the input image into m object patches, \mathcal{E}_{img}(\cdot) represents CLIP’s image encoder that encodes each object patch into a c -dimensional semantic vector (typically c=512 ), and \textbf{B}^{(i)(j)}_{s}\in\mathbb{R}^{c} is the semantic vector for the j -th object in the i -th view. Meanwhile, we obtain a depth map for each view via 3DGS depth rendering, and aggregate these to build a DB \textbf{B}_{d}\in\mathbb{R}^{n\times H\times W} for storing the scene’s depth information. This step can be formulated as: \textbf{B}^{(i)}_{d}=\mathcal{D}(c_{i};\textbf{G}) (3) where \mathcal{D}(\cdot;\textbf{G}) denotes the depth rendering of the 3D Gaussian scene G , c_{i} represents the camera parameters for the i -th view, and \textbf{B}^{(i)}_{d}\in\mathbb{R}^{H\times W} is the depth map for the i -th view, which has the same resolution to view v_{i} . 2.2 (B) Query: Retrieval For Localizing (RFL) As shown in Fig. 1 (B), this stage is for retrieving relevant patches of views for a given textual query and localizing objects within those clues. First, we encode the text query Q (such as “a red apple”) into a semantic vector using CLIP’s text encoder. We then compute the similarity between this vector and each item of SVB \textbf{B}_{s} , selecting the patch with the highest similarity in each view. This allows us to obtain the 2D localization L_{2D} of the target object in each view. This step can be formulated as: L^{(i)}_{2D}=\operatorname{position}(\arg\max_{j}\mathcal{S}(\mathcal{E}_{text}(Q),\textbf{B}^{(i)}_{s})) (4) where \mathcal{E}_{text} denotes text encoder, \mathcal{S}(\cdot,\cdot) denotes the cosine similarity function, and L^{(i)}_{2D}\in\mathbb{R}^{2} represents the 2D coordinates of the target object in the i -th view. By incorporating depth information, we can map the 2D location L^{(i)}_{2D} to 3D space, thereby obtaining the 3D position L^{(i)}_{3D} of the target object. This step can be formulated as: L^{(i)}_{3D}=\mathcal{P}^{-1}(L^{(i)}_{2D},\textbf{B}^{(i)}_{d},c_{i}) (5) where \mathcal{P}^{-1}(\cdot,\cdot,\cdot) denotes the back-projection function that maps 2D coordinates to 3D space using the depth map and camera parameters, and L^{(i)}_{3D}\in\mathbb{R}^{3} represents the 3D

Read the complete text