GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering

Saumya Saxena, Blake Buchanan, Chris Paxton, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, Oliver Kroemer · 2024

Read GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering by Saumya Saxena, Blake Buchanan, Chris Paxton, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, Oliver Kroemer (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14480.txt.

Introduction

Saumya Saxena and co-authors present a 2024 system that lets a robot explore a new space and answer questions about it, called GraphEQA. The robot must build its own grasp of a place it has never seen, then answer a question tied to that place with care. Weak scene records and slow search make the task hard.

The system keeps a live 3D map labeled with what each part is, plus photos tied to the task, as its memory. A language-vision model reads that memory to answer. Planning runs in layers that mirror the map's own layers, and the labels in the map steer where the robot looks next. On two test sets the system beat leading rivals, answering more tasks right with fewer planning steps. Trials in real homes and offices backed the design.

Catalog record

CREATORSaumya Saxena, Blake Buchanan, Chris Paxton, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, Oliver Kroemer
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering Thanks: This work was in part supported by the National Science Foundation under Grant No. CMMI-1925130 and in part by the EU Horizon 2020 research and innovation program under grant agreement No. 101017274 (DARKO). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Saumya Saxena, Blake Buchanan, Chris Paxton, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, Oliver Kroemer Abstract In Embodied Question Answering (EQA), agents must explore and develop a semantic understanding of an unseen environment in order to answer a situated question with confidence. This remains a challenging problem in robotics, due to the difficulties in obtaining useful semantic representations, updating these representations online, and leveraging prior world knowledge for efficient exploration and planning. Aiming to address these limitations, we propose GraphEQA, a novel approach that utilizes real-time 3D metric-semantic scene graphs (3DSGs) and task relevant images as multi-modal memory for grounding Vision-Language Models (VLMs) to perform EQA tasks in unseen environments. We employ a hierarchical planning approach that exploits the hierarchical nature of 3DSGs for structured planning and semantic-guided exploration. Through experiments in simulation on the HM-EQA dataset and in the real world in home and office environments, we demonstrate that our method outperforms key baselines by completing EQA tasks with higher success rates and fewer planning steps. Videos and code are available on our website . Figure 1 : Overview of GraphEQA: A novel approach for utilizing real-time 3D metric-semantic hierarchical scene graphs and task-relevant images as multimodal memory for grounding vision-language based planners to solve embodied question answering tasks in unseen environments. 1 Introduction Embodied Question Answering (EQA) Das et al. 2018 is a challenging task in robotics, wherein an agent is required to explore and understand a previously unseen environment sufficiently well, in order to answer an embodied question in natural language. Accomplishing this task efficiently requires agents to rely on both commonsense knowledge of human environments as well as ground its exploration strategy in the current environment context. For example, to answer the question “How many chairs are there at the dining table?”, the agent might rely on commonsense knowledge to understand that dining tables are often associated with dining rooms and dining rooms are usually near the kitchen towards the back of a home. A reasonable navigation strategy would involve navigating to the back of the house to locate a kitchen. To ground this search in the current environment, however, requires the agent to continually maintain an understanding of where it is, memory of where it has been, and what further exploratory actions will lead it to relevant regions. Finally, the agent needs to observe the target object(s) and perform visual grounding, in order to reason about the number of chairs around the dining table, and confidently answer the question correctly. Maintaining a concise and effective memory and using it to take actions in the environment is critical for solving EQA tasks. Prior works have demonstrated the impressive commonsense reasoning capabilities of Vision Language Models (VLMs) as planning agents, while leveraging a semantic map for retrieval Gu et al. 2023 or semantic exploration Ren et al. 2024 . In such approaches, the VLMs are not grounded in the current environment and the tasks of commonsense reasoning and context-based decision-making are disconnected. Recent works Rana et al. 2023 ; Shah et al. 2024 ; Anwar et al. 2024 ; Xie et al. 2024 ; Xu et al. 2024b focus on maintaining memory modules that can be queried by VLM agents for grounded planning. To construct a semantically rich memory, prior works either maintain a large set of images (not compact) Xu et al. 2024b ; Majumdar et al. 2024 or have to perform an expensive offline processing step to obtain a compact representation Anwar et al. 2024 ; Xie et al. 2024 ; Xu et al. 2024a . Thus, such semantic memory modules are either semantically rich Xie et al. 2024 ; Gu et al. 2023 ; Werby et al. 2024 , compact Rana et al. 2023 ; Shah et al. 2024 , or online Shah et al. 2024 —but not all at the same time. To address these limitations, we propose GraphEQA, an approach for building an online, compact, multimodal semantic memory that combines global, semantically-sparse, and task-agnostic information from real-time 3D scene graphs Hughes et al. 2022 with the local and semantically-rich information from task-relevant images Xu et al. 2024a . GraphEQA uses this multimodal representation for grounding vision-language planners to solve EQA tasks in unseen environments. Specifically, we utilize a recent spatial perception system Hughes et al. 2022 that incrementally creates a real-time 3D metric-semantic hierarchical scene graph (3DSG), given sequential egocentric image frames. We further augment this scene graph with semantic room labels and semantically-enriched frontiers. We also maintain a task-relevant visual memory that keeps track of task-relevant images as the robot explores the environment. Finally, we employ a hierarchical planning approach that utilizes the hierarchical nature of scene graphs and semantically relevant frontiers for structured planning and exploration in an unseen environment before using the multimodal memory to answer the embodied question with high confidence. We demonstrate that given our real-time multimodal memory and hierarchical planning approach, the agent is able to accomplish long-horizon tasks with significantly fewer VLM planning steps, explores explainable task-relevant frontiers, and succeeds in EQA tasks with a higher rate than previous approaches. We demonstrate our results on the HM-EQA dataset Ren et al. 2024 in the Habitat simulation environment Yadav et al. 2022 and also in the real world using the Hello Robot Stretch mobile manipulator in two indoor scenes. The key contributions of this work are as follows: • We present GraphEQA, a novel approach for using real-time 3D metric-semantic hierarchical scene graphs and task-relevant images as multimodal memory, for grounding VLMs to solve EQA tasks in unseen environments. • We propose an approach to enrich 3DSGs with 1) semantically enriched frontiers and 2) semantic room labels. • We propose a hierarchical VLM-based planning approach that exploits the hierarchical nature of the enriched 3DSG for structured exploration and planning. • We provide extensive experimental results, both in simulation on the HM-EQA dataset Ren et al. 2024 and in the real-world in two indoor environments, home and office, using the Hello Robot Stretch RE2 mobile manipulator. 2 Related Work 3D semantic scene graph representations for planning: 3D semantic scene graphs Armeni et al. 2019 ; Kim et al. 2020 ; Rosinol et al. 2021 ; Wald et al. 2020 ; Greve et al. 2024 have recently emerged as a compact, efficient, and semantically rich representation for indoor environments, leading to recent developments in both offline Werby et al. 2024 ; Gu et al. 2023 ; Li et al. 2024 ; Chang et al. 2023a and online scene graph generation methods Hughes et al. 2022 ; Maggio et al. 2024 ; Wu et al. 2021 . Offline methods Werby et al. 2024 ; Gu et al. 2023 focus on building open-vocabulary semantic scene graphs, by semantically enriching the nodes and edges of a scene graph (affordances

Read the complete text