SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman · 2026
Read SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching by Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21788.txt.
Introduction
Xuefei Sun and co-authors propose SceneGraphGrounder, a zero-shot visual grounding method that recasts grounding as graph matching. A marker prompting trick lets a vision-language model infer object-object links from views, which are lifted into a lasting scene graph holding spatial and meaning relations. A query graph is then aligned with the scene graph under rules, giving matching views and clear reasoning. Tests on ScanRefer show strong zero-shot scores from RGB-D inputs alone, and a robot field test shows robust spatial thinking over long runs.
Catalog record
| CREATOR | Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35861 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21788
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman Abstract Zero-shot 3D visual grounding requires localizing objects in unstructured environments from free-form natural language. Recent vision–language model (VLM) approaches achieve promising results but rely on view-dependent reasoning or implicit representations, limiting spatial consistency and interpretability for compositional queries. We propose SceneGraphGrounder, a framework that reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph. To enable this formulation, we introduce a visual marker prompting strategy that enables a VLM to infer object–object relationships from 2D views, which are subsequently lifted into a persistent 3D scene graph encoding both spatial and semantic relations. Given a query, we construct a query graph and perform constrained alignment with the scene graph, ensuring multi-view consistency and interpretable reasoning. Experiments on the ScanRefer benchmark demonstrate that our method achieves competitive performance among zero-shot approaches, using only RGB-D inputs. We further validate our framework through real-world deployment on a mobile robot, demonstrating robust spatial reasoning in long-horizon physical environments. We will make our code publicly available upon acceptance. I Introduction 3D visual grounding refers to localizing a target object in a 3D scene using a free-form natural language description provided by a human. The task becomes particularly challenging in zero-shot and open-vocabulary settings, where object categories and spatial relations are not constrained to predefined labels. In these scenarios, models must interpret diverse object references, compositional descriptions, and nuanced spatial cues, demanding a deeper and more flexible form of semantic understanding beyond fixed label spaces. Recent approaches to address these challenges have increasingly turned to large vision-language models (VLMs) and large language models (LLMs). In particular, agent-based frameworks formulate grounding as an iterative reasoning process over multi-view observations or external tools [ 31 , 29 ] . In parallel, alternative methods seek to bridge 3D scenes with 2D VLMs via rendering and viewpoint adaptation, enabling powerful image-based models to operate on 3D data [ 13 , 35 ] . While these approaches have achieved strong performance, they largely rely on implicit scene representations, reasoning directly over rendered views, proposal features, or intermediate tool outputs. As a result, spatial relationships are inferred on-the-fly rather than explicitly stored, leading to potential inconsistencies across viewpoints and requiring repeated reasoning for each query, which limits efficiency and prevents reuse of scene structure. The lack of an explicit relational representation further reduces interpretability and weakens robustness on compositional queries involving multiple objects and spatial constraints. These limitations raise a fundamental question: how should a 3D environment be represented for grounding and language reasoning? Prior work explores a spectrum of representations, from CLIP-aligned semantic maps [ 11 , 17 ] to 3D hierarchical scene graph [ 20 ] . At the same time, natural language descriptions are inherently relational and compositional, and they can be naturally represented as graphs [ 2 ] , where entities correspond to nodes and semantic relations define edges. Similarly, in the robotics field, grounding language can also be formulated as graphs [ 16 ] , enabling compositional interpretation through graph-based alignment between linguistic structure and the world. This shared graph-like structure of both language queries and 3D scenes motivates our approach: by explicitly constructing a graph for the scene and aligning it with a graph derived from the language query, grounding can be reformulated as a structured matching problem rather than relying on repeated implicit reasoning at inference time. In this work, we present SceneGraphGrounder , a framework that formulates 3D visual grounding as structured graph matching between a query graph and a 3D scene graph. We represent queries as graphs and perform grounding via graph matching to a semantically rich 3D scene graph constructed from RGB-D data, augmented with VLM-inferred object relationships transferred from 2D views. The primary contributions of this work are: • Formulate 3D visual grounding as a structured graph matching problem between a query graph and a reconstructed 3D scene graph, improving efficiency and interpretability. • Construct a semantically rich 3D scene graph that encodes both spatial and semantic relationships, enabling structured and reusable relational reasoning, to support graph matching. • Enable zero-shot 3D visual grounding from raw RGB-D observations and odometry, without ground-truth point clouds or precomputed object proposals. Our work achieves competitive performance with prior zero-shot methods that rely on privileged 3D inputs. II Related Works Fig. 1 : System Overview. Our framework takes RGB-D images, sensor odometry, and a user query as input. The graph is lifted into 3D via visual prompting and association with the reconstructed point cloud, forming a global 3D scene graph. Object grounding is achieved via graph matching with the parsed query. II-A Scene Graph Generation 2D scene graph generation (SGG) has increasingly shifted toward VLM-based formulations, where the task is reframed as multimodal reasoning or generation over object and relationship [ 28 , 34 ] . Recent open-vocabulary approaches further improve generalization to unseen objects and predicates by leveraging large VLMs and language priors. Within this line of work, early methods primarily rely on prompt-based designs, whereas more recent approaches shift toward zero-shot alignment [ 5 , 28 ] , probabilistic grounding [ 15 ] , and distillation strategies [ 4 ] built on pretrained models such as CLIP [ 19 ] . Recent research has extended SGG beyond 2D to 3D environments using point clouds or multi-view RGB images, broadly categorized into methods leveraging explicit 3D geometry for contextual reasoning [ 21 , 24 , 37 ] and those operating on multi-view images, sometimes enhanced with depth or SLAM-based reconstruction [ 26 , 25 ] . Prior work explores pairwise relationship modeling [ 21 ] , GCN-based aggregation and edge-centric reasoning [ 26 , 33 ] , as well as integration of commonsense knowledge and external priors [ 38 , 7 ] , or knowledge distillation from multimodal teachers [ 24 ] . Despite these advances, most approaches still focus on constructing a global scene graph and struggle to incorporate structured, task-driven reasoning. II-B 3D Visual Grounding 3D visual grounding aims to localize objects in 3D scenes based on natural language descriptions. Early methods [ 3 , 1 ] relied on supervised learning with aligned 3D annotations and fixed object categories, typically using point cloud encoders or proposal-based frameworks. While effective in closed-set settings, their generalization to unseen categories and compositional expressions is limited. Recent open-vocabulary approaches leverage vision–language representations to align text with 3D object proposals [ 40 , 36 , 17 ] , but they often rely on object-level similarity and lack explicit relational reasoning, limiting robustness for complex expressions. Recent transformer-based approaches improve global multimodal fusion, and this line of work is further extended by methods such as [ 13 ] , which leverages multi-view reasoning and VLM integration to achieve strong zero-shot and open-vocabulary performance. Building on similar ideas, agent-based frameworks [ 31 , 29 ] decompose language queries and coordinate perception tools or multi-view reasoning. In a related direction, methods have been proposed that bridge 3D scenes with 2D VLMs through rendering-based and modular reasoning strategies [ 13 , 35 ] . Despite their effectiveness, these approaches typically rely on implicit scene representations and perform relational