Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces

Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, Francis Engelmann · 2025

Read Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces by Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, Francis Engelmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19199.txt.

Introduction

Chenyangguang Zhang, Alexandros Delitzas, and five colleagues build three-dimensional scene graphs that record what objects are for, not just where they sit, in a paper published in two thousand twenty-five. A functional scene graph notes interactive elements, a handle, a switch, a lid, and how they relate to their objects and to each other.

No training set existed for this, so the method leans on foundation models: vision-language models and language models supply the functional knowledge. Inputs are posed color and depth images of real indoor rooms. The authors evaluate on an extended public dataset and a new one they collected, both annotated with functional graphs. Their system clearly beats adapted baselines built for spatial scene graphs. Downstream, the graphs support three-dimensional question answering about a room. A robot that knows which part opens the cabinet is a step beyond one that merely sees a cabinet.

Catalog record

CREATORChenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, Francis Engelmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, Francis Engelmann Abstract We introduce the task of predicting functional 3D scene graphs for real-world indoor environments from posed RGB-D images. Unlike traditional 3D scene graphs that focus on spatial relationships of objects, functional 3D scene graphs capture objects, interactive elements, and their functional relationships. Due to the lack of training data, we leverage foundation models, including visual language models (VLMs) and large language models (LLMs), to encode functional knowledge. We evaluate our approach on an extended SceneFun3D dataset and a newly collected dataset, FunGraph3D, both annotated with functional 3D scene graphs. Our method significantly outperforms adapted baselines, including Open3DSG and ConceptGraph, demonstrating its effectiveness in modeling complex scene functionalities. We also demonstrate downstream applications such as 3D question answering and robotic manipulation using functional 3D scene graphs. See our project page at https://openfungraph.github.io . 1 Introduction Posed RGB-D Frames Functional 3D Scene Graph Fig. 1 : Functional 3D Scene Graphs. Given an input sequence of posed RGB-D frames of an indoor environment, our method predicts a functional 3D scene graph by detecting objects, identifying interactive elements, and inferring functional relationships. This enables the representation of interactions, functions, and scene dynamics, going beyond existing 3D scene graph methods that are constrained to spatial relationships between static objects. This paper introduces functional 3D scene graphs for real-world indoor spaces from posed RGB-D images. 3D scene graphs offer a lightweight, abstract representation for capturing the comprehensive semantic structure of an environment [ 4 ] . They support a variety of applications, including 3D scene alignment [ 66 ] , image localization [ 51 ] , graph-conditioned 3D scene generation [ 97 , 21 ] , as well as robotics navigation [ 83 ] and task planning [ 2 , 61 ] . Recent advances in 3D scene graph prediction [ 11 , 41 , 27 , 4 , 40 , 84 , 63 , 64 , 78 ] , have enabled exciting developments across multiple areas, including scene graph inference from 3D reconstructions [ 78 , 11 ] , applications for robotic interactions [ 27 , 84 ] , online scene graph generation [ 84 ] , open-vocabulary 3D scene graphs [ 40 , 41 ] and large-scale, hierarchical scene graphs [ 4 , 63 , 64 ] . The performance of recent scene graph methods also benefits from advancements in 3D scene understanding techniques [ 14 , 57 , 70 ] , which they rely on to extract objects and their semantics for modeling inter-object relationships. However, existing 3D scene graph estimation methods [ 78 , 40 , 27 , 84 ] face important limitations: graph nodes are typically restricted to objects , and edges represent only spatial relationships. For instance, edges primarily capture relative positions, such as ‘the TV is mounted on the wall’ or ‘the flower is placed on the table’ —information already implicitly encoded by object positions. Crucially, these methods lack representations of small interactive elements [ 17 ] and their functional relationships with other scene objects, which are essential for finer-grained interactions ( e.g . , flipping a switch to turn on a light), making them less suitable for higher-level functional reasoning . The key idea of this paper is to enhance 3D scene graphs with the capability to represent functional relationships between objects and their interactive elements. A 3D scene graph that captures both functionalities and interactions opens up significant opportunities. For example, robotic agents can identify interactive elements and their functional relationships with objects to perform effective manipulation tasks, or graph-guided 3D scene generation methods [ 97 , 21 ] can, with this enriched representation, generate more dynamic and realistic environments by incorporating interactive elements and their effects. However, creating functional 3D scene graphs is challenging. Most importantly, there is a lack of training data to learn the complex functional relationships between objects and their interactive elements. Unlike existing 3D scene graphs, functional 3D scene graphs require a more nuanced understanding of interactions and object affordances. To address this, our approach implements an open-vocabulary pipeline for functional 3D scene graph inference, termed OpenFunGraph , leveraging the extensive knowledge encoded within foundation models, including visual language models (VLM) and large language models (LLM). These models, pre-trained on vast amounts of multimodal data, include rich semantic information that can potentially be adapted for functional understanding. This leads us to the central question of this work: “Can we harness foundation models to construct functional 3D scene graphs?” We evaluate our approach on two challenging datasets: an extended version of SceneFun3D [ 17 ] with newly added functional relationship annotations, and FunGraph3D, a freshly collected real-world dataset featuring high-precision 3D laser scans, accurately registered To address these limitations, we introduce functional 3D scene graphs, which model objects, interactive elements, and their functional relationships within a unified structure (formally defined in Section 3 ). This representation extends traditional 3D scene graphs by incorporating interactive sub-parts alongside objects and representing functional relationships beyond simple spatial ones. We argue that functional 3D scene graphs should possess the following characteristics. First, the representation should operate in an open-vocabulary manner to enhance generalization and applicability. Second, it should be flexible , allowing various attributes to be attached to nodes ( e.g . , sensor data, natural language captions, semantic features) and edges ( e.g . , relationship descriptions), thus ensuring adaptability for downstream applications. In summary, our key contributions are: • We introduce functional 3D scene graphs that extend traditional 3D scene graphs by capturing functional relationships between objects and interactive elements. • We propose a novel approach that leverages the knowledge embedded in foundation models, specifically VLMs and LLMs, to construct functional 3D scene graphs without task-specific training. • We present a new real-world dataset, FunGraph3D, with ground-truth functional annotations, and demonstrate that our method outperforms adapted baselines, including Open3DSG and ConceptGraph. Node Detection (Sec. 4.1 ) Node Description (Sec. 4.2 ) Functional Edges (Sec. 4.3 ) [formula] [formula] [formula] Fig. 2 : Illustration of the OpenFunGraph architecture. Given a sequence of posed RGB-D frames \{(\mathcal{I}_{i},\mathcal{D}_{i})\}_{i=1}^{n} , we use RAM++ [ 104 ] and GroundingDINO [ 49 ] to detect and segment objects {\color[rgb]{0,0.5,1}\boldsymbol{\mathcal{O}}}{} and interactive elemens {\color[rgb]{0,0.88,0}\boldsymbol{\mathcal{I}}}{} , forming the node candidates of the functional 3D scene graph. Next, a mechanism using the large language model (LLM) GPT [ 1 ] and the visual language model (VLM) LLAVA [ 48 ] generates natural language descriptions \mathcal{L} for each node. Finally, we infer functional relationships {\color[rgb]{1,0.5,0}\boldsymbol{\mathcal{R}}}{} between objects {\color[rgb]{0,0.5,1}\boldsymbol{\mathcal{O}}}{} and interactive elements {\color[rgb]{0,0.88,0}\boldsymbol{\mathcal{I}}}{} , represented as the edges in the functional 3D scene graph {\color[rgb]{0.72,0,0}\boldsymbol{\mathcal{G}}}{} . 2 Related Work 3D indoor scene understanding. Many works concentrate on closed-set 3D semantic segmentation [ 5 , 14 , 31 , 32 , 33 , 42 , 80 , 45 , 57 , 58 , 76 , 81 ] or instance segmentation [ 23 , 28 , 29 , 74 , 38 , 70 , 77 , 96 ] on the existing 3D indoor scene understanding benchmarks [ 15 , 10 , 37 , 72 , 65 , 3 , 7 , 93 ] .

Read the complete text