Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning
2025
Read Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05894.txt.
Introduction
Fei Yu and co-authors build scene graphs of 3D spaces that robots can query in plain language. A scene graph lists objects and their relations. Open-world settings break the fixed label sets such systems usually need, since new objects and relations appear freely.
Their framework pairs vision-language models with retrieval-based reasoning. One module builds the graph on the fly, detecting objects and inferring relations without a closed vocabulary. The graphs are stored in a vector database, so text or image queries can pull relevant scenes for question answering, grounding, retrieval, and task planning. On two benchmarks the method generalized well and beat rivals. The authors pitch it as a scalable route to interactive 3D understanding.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05894
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning Thanks: Correspondding Author: [email protected] Abstract Understanding 3D scenes in open-world settings poses fundamental challenges for vision and robotics, particularly due to the limitations of closed-vocabulary supervision and static annotations. To address this, we propose a unified framework for Open-World 3D Scene Graph Generation with Retrieval-Augmented Reasoning, which enables generalizable and interactive 3D scene understanding. Our method integrates Vision-Language Models (VLMs) with retrieval-based reasoning to support multimodal exploration and language-guided interaction. The framework comprises two key components: (1) a dynamic scene graph generation module that detects objects and infers semantic relationships without fixed label sets, and (2) a retrieval-augmented reasoning pipeline that encodes scene graphs into a vector database to support text/image-conditioned queries. We evaluate our method on 3DSSG and Replica benchmarks across four tasks—scene question answering, visual grounding, instance retrieval, and task planning—demonstrating robust generalization and superior performance in diverse environments. Our results highlight the effectiveness of combining open-vocabulary perception with retrieval-based reasoning for scalable 3D scene understanding. Introduction Understanding 3D scenes is fundamental for tasks like autonomous navigation and augmented reality. However, most traditional methods rely on closed-vocabulary annotations and curated datasets, limiting their scalability in dynamic, unstructured environments where novel objects and interactions frequently emerge. This gap calls for open-world 3D understanding methods that can generalize to unseen scenarios without exhaustive supervision. Recent advances in vision-language models (VLMs) such as CLIP ( 9 ) , ALIGN ( 15 ) , and ImageBind ( 6 ) have enabled open-vocabulary learning, which shows remarkable potential in 2D tasks like classification and detection. These successes have inspired efforts to adapt VLMs for 3D vision, leading to progress in tasks such as semantic segmentation and object recognition. However, these methods often rely on 2D-3D projections and RGB-D inputs with known poses, which are impractical in many real-world settings due to sensor limitations, occlusions, and viewpoint variations. To address these limitations, we introduce a retrieval-augmented open-world 3D scene graph generation framework. It consists of two core components: (1) a dynamic scene graph module that constructs semantic and spatial representations of the environment by identifying objects and relationships without fixed label sets; and (2) a retrieval-augmented navigation system that enables natural language-based exploration of 3D scenes. This work is motivated by the need for adaptive and scalable 3D understanding systems that can operate in open-world settings without extensive human supervision. While conventional methods perform well in constrained environments, they struggle to generalize to dynamic and diverse scenarios. By integrating VLMs with retrieval mechanisms, our method enables compositional reasoning over unseen objects and relationships. Our framework addresses the following challenges: (1) Open-Vocabulary Grounding : Recognizing and localizing unseen objects and relations without predefined labels. (2) Dynamic Graph Construction : Building and updating 3D scene graphs that capture evolving semantic and spatial relationships. (3) Language-Guided Interaction : Supporting navigation and querying through natural language, requiring alignment between textual and spatial modalities. Our contributions are summarized as follows: • A novel framework for open-world 3D scene graph generation, leveraging vision-language models to extract objects and relationships without fixed annotations. • A retrieval-augmented reasoning module that enables flexible scene interaction through query-based exploration. • Extensive experiments on 3DSSG and Replica benchmarks demonstrating improved performance and generalization in dynamic, real-world environments. Related Work Open-Vocabulary Scene Understanding. The success of 2D vision-language models (VLMs) such as CLIP ( 9 ) , ALIGN ( 15 ) , and ImageBind ( 6 ) has spurred interest in extending open-vocabulary capabilities ( 29 ; 27 ) to 3D tasks, including semantic and instance segmentation ( 8 ; 10 ; 36 ) . These early efforts often fuse CLIP-based supervision with 3D backbones using RGB-D data with known poses, but their performance is limited by projection errors, occlusion artifacts, and strong reliance on pose priors. Open-Vocabulary Scene Graph Generation. 3D scene graph generation aims to model both semantic and spatial relationships among objects, serving as a bridge between visual perception and symbolic reasoning. Prior work such as ( 28 ; 24 ) focused on structured indoor data with supervised labels. Recent methods like Open3DSG ( 17 ) and zero-shot variants ( 22 ) leverage VLMs to recognize unseen objects and infer spatial relations. However, many still depend on annotated RGB-D inputs or fixed camera poses, which limits their adaptability to open-world settings. In contrast, our method eliminates the need for manual annotations or fixed-pose RGB-D data, enabling fully annotation-free 3D scene graph generation. It achieves strong performance in both open- and closed-vocabulary benchmarks while offering better generalization to unseen environments. Efficient Multimodal Large Language Models Adaptation. Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for enabling joint reasoning over visual and textual inputs ( 33 ; 13 ; 3 ; 31 ; 37 ; 30 ; 21 ; 25 ) . These models typically comprise three components: a large language model (LLM) backbone (e.g., LLaMA ( 5 ) , Qwen ( 2 ) ), a visual encoder (e.g., CLIP ( 23 ) , BLIP ( 19 ) ), and adapter modules that align visual features with the LLM’s embedding space ( 35 ) . This architecture supports a wide range of vision-language tasks, such as image captioning, visual question answering, and visual grounding. Given the size and complexity of MLLMs, training them from scratch is computationally expensive ( 16 ) . To address this, Parameter-Efficient Fine-Tuning (PEFT) methods ( 11 ) —such as LoRA ( 12 ) and Prefix Tuning ( 20 ) —have been developed to adapt MLLMs to downstream tasks by updating only a small subset of parameters. These approaches significantly reduce memory and computational costs, making MLLMs more deployable in real-world settings. However, challenges remain in adapting to long-tail or domain-specific scenarios where fine-grained cross-modal alignment is required. Recent research highlights the need for improved adaptation strategies that preserve generalization while enhancing task-specific performance ( 12 ; 20 ) . Methodology Figure 1: Overview of the proposed framework for Open-World 3D Scene Graph Generation and Retrieval-Augmented Navigation. The framework comprises two key components: (1) a 3D Scene Graph Generator that incrementally builds semantic and spatial representations from RGB-D sequences by detecting objects, estimating poses, selecting optimal viewpoints, and extracting inter-object relations via vision-language reasoning; and (2) a Retrieval-Augmented Reasoning module that transforms the scene graph into a vectorized knowledge base to support three categories of interaction: (i) spatial object queries, (ii) semantic relationship reasoning, and (iii) instance-level retrieval. This integrated design enables grounded, multimodal, and context-aware interaction within dynamic open-world 3D environments. To enable robust navigation in open-world environments, our approach builds on retrieval-augmented reasoning over 3D scene graphs, which encode object instances, spatial relations, and high-level semantics. Unlike closed-world settings with fully known and static layouts, open-world scenarios introduce challenges such as dynamic object placement, occlusion, and real-time scene changes. These issues require scene representations that are both adaptive and incrementally updatable. To address this, we propose a retrieval-augmented framework that leverages previously stored scene graphs to support inference and decision-making under partial observations. By retrieving relevant spatial-semantic context from past interactions, the agent can recover missing structures, disambiguate uncertain observations, and guide navigation in dynamically evolving environments. Definition 1: A 3D scene graph provides a structured graph-based representation