GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering

Muhammad Qasim AliUniversity of Waterloo [email protected], Saeejith NairUniversity of Waterloo [email protected] Alexander WongUniversity of Waterloo [email protected] Yuchen CuiUniversity of California, Los Angeles [email protected] Yuhao ChenUniversity of Waterloo yuhao.chen1@uwaterloo · 2025

Read GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering by Muhammad Qasim AliUniversity of Waterloo [email protected], Saeejith NairUniversity of Waterloo [email protected] Alexander WongUniversity of Waterloo [email protected] Yuchen CuiUniversity of California, Los Angeles [email protected] Yuhao ChenUniversity of Waterloo yuhao.chen1@uwaterloo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01174.txt.

Introduction

Muhammad Qasim Ali and colleagues give a household robot a writable memory for answering questions about its surroundings. Robots usually freeze a three-dimensional scene map before the task starts, so a new question can miss objects or relations nobody recorded. Writing in twenty twenty-five, their system keeps three parts. A scene graph the agent can edit through software calls, a navigation log indexing what each camera frame showed, and a scratchpad for task notes. The agent updates the map in words as the task unfolds, rather than rebuilding it from scratch.

On a standard embodied question answering benchmark, the editable memory beat an image-only baseline using the same vision-language model. It won by a few points of accuracy, while using five times fewer frames. The authors argue that online, language-driven upkeep makes scene memory both leaner and more useful.

Catalog record

CREATORMuhammad Qasim AliUniversity of Waterloo [email protected], Saeejith NairUniversity of Waterloo [email protected] Alexander WongUniversity of Waterloo [email protected] Yuchen CuiUniversity of California, Los Angeles [email protected] Yuhao ChenUniversity of Waterloo yuhao.chen1@uwaterloo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering Muhammad Qasim AliUniversity of Waterloo [email protected], Saeejith NairUniversity of Waterloo [email protected] Alexander WongUniversity of Waterloo [email protected] Yuchen CuiUniversity of California, Los Angeles [email protected] Yuhao ChenUniversity of Waterloo [email protected] Abstract Structured scene representations are a core component of embodied agents, helping to consolidate raw sensory streams into readable, modular, and searchable formats. Due to their high computational overhead, many approaches build such representations in advance of the task. However, when the task specifications change, such static approaches become inadequate as they may miss key objects, spatial relations, and details. We introduce GraphPad , a modifiable structured memory that an agent can tailor to the needs of the task through API calls. It comprises a mutable scene graph representing the environment, a navigation log indexing frame-by-frame content, and a scratchpad for task-specific notes. Together, GraphPad serves as a dynamic workspace that remains complete, current, and aligned with the agent’s immediate understanding of the scene and its task. On the OpenEQA benchmark, GraphPad attains 55.3 % accuracy— +3.0 pp over an image‑only baseline using the same vision–language model—while operating with five times fewer input frames. These results show that allowing online, language‑driven refinement of 3‑D memory yields more informative representations without extra training or data collection. 1 Introduction A household robot has just scanned the kitchen when a user asks, “Is the red mug back inside the upper cupboard?” Because the scene graph was built from only a handful of earlier keyframes, it lacks both a node for the mug and the relation linking it to the cupboard interior. With this critical information missing, the agent must either guess, rescan the entire scene, or refuse the query altogether. This scenario highlights a fundamental limitation in current embodied AI: structured 3D memories are typically finalized before the downstream task is known, frequently omitting objects and spatial relations that later prove essential for action or reasoning . Inherent limitations of static memories. Modern embodied systems commonly pair a vision-language model (VLM) with pre-computed structured scene representations such as 3D scene graphs Hughes et al. 2022 ; Li et al. 2024 . These structured scene representations help to consolidate large amounts of dense, noisy RGB-D frames into a shared scene representation. By modeling the scene as distinct entities and relations, Scene Graphs help VLMs structure their complex multi-step spatial reasoning while performing navigation and manipulation tasks Li et al. 2024 ; Chiang et al. 2024 . A core challenge with such memory systems is that crucial compression decisions—specifically, which objects to retain and what aspects of the scene to describe—are made once. When the subsequent task shifts, these pre-configured, static representations can become insufficient, incomplete, or inaccurate for the new task at hand. This fundamental flaw is vividly demonstrated in Embodied Question Answering (EQA) Das et al. 2018 . Here, generic scene graphs, built without any foresight into the specific questions that will be asked, consistently perform poorly when attempting to provide answers Majumdar et al. 2024 . This underscores a critical point: a memory system optimized for one general purpose simply can’t adapt effectively to the nuanced demands of a different, more specific task. Current solutions follow two primary approaches. Over-provisioning retains nearly every candidate detection, incurring substantial memory and computational overhead Anwar et al. 2024 . Offline enhancement pipelines append heuristically chosen affordances or relations for anticipated tasks Li et al. 2024 , but each new domain demands additional engineering effort and cannot recover information never initially detected. Neither strategy scales effectively when user requests vary widely. Our approach: language-guided memory updates. We propose that an embodied agent should be able to dynamically revise its structured scene memory when reasoning reveals a gap in its knowledge. Rather than discarding an inadequate structured representation—and thereby losing its benefits—or recreating it from scratch at high computational cost, the agent should be able to modify and enhance the existing representation. This allows the agent to incorporate task-relevant information without starting over. Moreover, the agent should be able to do this through targeted perceptual queries on specific keyframes, and seamlessly integrate the newly discovered information into its scene graph. GraphPad. We introduce GraphPad , a modifiable 3D scene graph memory whose content can be updated at inference time through language-level commands issued by the same VLM that performs high-level reasoning. Whenever uncertainty arises or missing information is identified, the VLM can: (i) retrieve additional keyframes from its navigation log, (ii) insert previously undetected objects or spatial relationships, and (iii) annotate existing nodes with task-specific attributes. These operations are exposed as language-callable functions that the reasoning agent itself can invoke, eliminating the need for task-specific reasoning code. 1 1 1 Generic vision modules—object detector, mask extractor, depth back-projection—remain necessary but are not specialized for any particular downstream task. Through successive targeted updates, the graph evolves from a coarse initial representation into a task-conditioned model containing precisely the information needed for the current query or action planning. Contributions. • We formulate language-driven online editing of structured 3D memories as a solution to the task–memory mismatch inherent in fixed scene graphs. • We present GraphPad , a system in which a single VLM both identifies knowledge gaps and updates its 3D representation via language function calls during inference. • On the OpenEQA benchmark Majumdar et al. 2024 , we demonstrate that GraphPad improves spatial question answering from 52.3% to 55.3% while reducing the number of processed frames from 25 to 5, outperforming both image-only and static-graph baselines without additional training. By recasting perception as an interactive, language-mediated process rather than a static preprocessing step, GraphPad enables more efficient and effective reasoning about complex 3D environments—a critical capability for agents that must bridge language understanding with physical action. Figure 1 : Overview of GraphPad for embodied question answering. The top row illustrates the Structured Scene Memory (SSM) components: Scene Graph with associated Graphical Scratch Pad, Frame Memory containing sparse keyframes, and Navigation Log indexing frame metadata. The bottom row demonstrates the inference process: given the question about a white object above the TV, the VLM first examines the initial memory state (left), identifies missing information and calls the analyze_frame API on a promising frame (middle), then integrates the newly detected air conditioner into the scene graph and scratch pad before providing the answer (right). This process enables dynamic, task-specific memory updates without exhaustive preprocessing. 2 Background and Related Work Current embodied AI systems struggle with a fundamental limitation: their memory representations often lack critical information needed during task execution. This section examines key research areas relevant to this challenge. 2.1 Embodied Memory Representations Frame-level memories store raw RGB-D streams with learned descriptors. Systems like ReMEmbR Anwar et al. 2024 and Embodied-RAG Xie et al. 2024 index thousands of images, allowing retrieval of relevant observations during inference. While flexible, these approaches face computational overhead from processing large image collections and delegate all spatial reasoning to language models. Semantic metric maps address these limitations by embedding visual features within geometric reconstructions. VL-Maps Huang et al. 2023 , PLA Ding et al. 2023 , and OpenScene Peng et al. 2023 support spatial grounding but lack explicit object boundaries and relationships essential for symbolic reasoning. 3D

Read the complete text