ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration

Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim · 2026

Read ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration by Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24068.txt.

Introduction

Lee and colleagues give exploring robots a memory organised like the places they move through. A robot in an unfamiliar building must notice what a task needs, spot what it has not seen yet, and go looking for the missing pieces. Their 2026 system, ObsGraph, stores observations in a three-layer graph: rooms as broad anchors, views capturing which objects appear together, and objects holding fine detail.

Retrieval runs coarse to fine under a fixed budget, starting at room level and narrowing only where evidence is thin. Crucially, what retrieval fails to find steers the next exploration step, choosing between searching a new room, refining a view, or heading to an unexplored frontier. Tests on embodied reasoning and exploration benchmarks show better success and efficiency than less structured approaches. Lee's team credits the tight loop between remembering, retrieving, and deliberately gathering the information still missing.

Catalog record

CREATORTaekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim Abstract Embodied reasoning and exploration are increasingly considered crucial abilities for robots operating in complex and unfamiliar environments. To accomplish tasks in such settings, an agent must identify and acquire the information necessary for the task through exploration. We propose ObsGraph, an observation-centric hierarchical scene graph that unifies scene representation, retrieval, and exploration. It retains visual evidence and organizes it into room–view–object layers: rooms provide coarse semantic anchors, views preserve contextual object co-visibility, and objects store fine-grained details. On top of this representation, we perform coarse-to-fine hierarchical retrieval under a bounded budget, and crucially use retrieval outcomes to structure the exploration candidate space—activating room-level exploration, view refinement, or frontier exploration—thereby tightly coupling representation, retrieval, and adaptive multi-scale exploration. Experiments across embodied reasoning and exploration benchmarks demonstrate improved success and efficiency, highlighting the benefits of structured scene representation and more targeted information gathering driven by identified evidence gaps. 1 Introduction Autonomous robots are expected to operate as versatile assistants in human environments, with stronger capabilities for perception [ 36 , 19 , 2 ] and interaction [ 1 , 5 , 40 ] . Both of these capabilities often assume the targets are fully observable. However, in the real world, robots are often not aware of the environment, and they should move and search to find relevant targets to meet task specifications. To complete such tasks efficiently and reliably, robots must reason about what is known, identify what is missing, and actively gather the necessary information. In this context, embodied reasoning and exploration are crucial, especially in complex environments. This capability refers to a robot’s ability to actively explore a scene, reason over accumulated observations, and identify missing information required for task completion. Its effectiveness fundamentally depends on scene representation, namely, how the agent organizes and maintains observations over time. As tasks vary, including multiple objectives within the same environment, the representation should be structured in a task-agnostic manner while preserving diverse evidence and supporting flexible access to task-relevant information. At the same time, the agent’s understanding of the environment is inherently partial, given that observations are limited by the exploration history. Therefore, the representation should make explicit what is currently known with respect to the given task and where additional observation is needed. Figure 1 : We propose ObsGraph, a unified framework that tightly integrates representation, retrieval, and exploration. It maintains an observation-centric scene graph that preserves rich visual evidence while structuring it hierarchically (room–view–object) to provide both spatial context and object co-visibility cues. Given a task, relevant information is retrieved across hierarchy levels, and the retrieved context naturally induces exploration candidates at corresponding scales. In this way, exploration is naturally structured by the representation itself. From these observations, an effective scene representation for embodied reasoning and exploration should satisfy the following properties. (1) It must preserve sufficient evidence while remaining compact, enabling reliable reasoning. (2) Given a task, it should allow flexible access to information at an appropriate level of abstraction and be structured to leverage contextual cues to accurately distinguish task-relevant information. (3) It should enable reasoning about what level of additional information would be most effective for progressing toward task completion. Dense 3D representations are a popular category and are advantageous for complex spatial understanding [ 12 , 37 ] . They embed visual information into their 3D geometry, preserving rich semantic information [ 22 , 32 , 17 ] . However, they lack explicit abstraction for task-conditioned access, making selective extraction of task-relevant regions nontrivial. Consequently, reasoning often must operate over large portions of the scene, which limits scalability as exploration expands. Scene graphs are another popular category due to their concise and structured format, which is well-suited for reasoning in complex environments [ 26 , 4 ] . Existing methods either summarize environments in an object-centric form by storing object properties and relations [ 10 , 29 ] or abstract them hierarchically [ 13 , 25 ] . Although these representations support efficient search at appropriate abstraction levels, they may lose task-relevant evidence due to premature abstraction before the task is specified. Recent methods instead perform reasoning directly over task-relevant image subsets [ 33 , 9 ] , preserving rich visual information without early abstraction. While effective for retaining detailed evidence, these approaches lack an explicit structural organization. Moreover, prior scene representations are primarily designed for storing current information about the environment, offering limited support for reasoning about what level of additional information is needed. Motivated by the trade-off between structured scene graphs and image-based reasoning, and by the need to reason about what to observe next, we propose an observation-centric scene graph. This representation preserves rich visual evidence and structurally grounds task-aware retrieval and exploration within a unified hierarchy. The proposed scene graph consists of three layers: room, view, and object. First, the room layer stores observed room types. It organizes views by room type, providing coarse cues to disambiguate similar information in lower layers. Second, the view layer stores selected observed images. It provides visual context, including relationships between objects and the background where each individual object is observed. We hypothesize that object context is determined by which other objects are observed together with it across views. Therefore, we design a view selection method that preserves diverse object co-visibility among observed images while avoiding visual redundancy. Third, the object layer stores object-centric crops for each object. It complements fine-grained visual evidence that cannot be easily captured by the view layer. Once the scene representation is updated, we perform hierarchical retrieval to assess whether sufficient task-relevant evidence (i.e., observations needed for the task) has been collected. It identifies task-relevant evidence at the appropriate abstraction level by leveraging both spatial context in the room–object hierarchy and semantic context in the view–object hierarchy. If evidence remains insufficient, we invoke our exploration strategy to jointly infer the missing task-relevant information and the exploration option that most effectively resolves it. For room-level retrieval, the agent uses room-level spatial cues and associated views to infer room exploration candidates that expand task-relevant room information. At the view level, it uses object and background context in retrieved views to infer view-refinement candidates (e.g., closer view, wider context, or different angle) that can acquire richer task-relevant evidence. Object-level evidence refine sufficiency assessment rather than to generate exploration candidates. Altogether, we unify representation, retrieval, and exploration by structuring the exploration option space according to the retrieved abstraction level in a single task-aware loop. In short, our contributions are as follows. • A task-agnostic, observation-centric scene graph that hierarchically organizes visual observations for task-aware reasoning and exploration • An informative view selection strategy for constructing a compact, context-rich view layer • A scene graph-guided coarse-to-fine retrieval scheme that jointly infers missing task-relevant information and effective exploration options using hierarchical structural and semantic context • An adaptive exploration strategy that adjusts exploration scale within the scene graph hierarchy based on identified task-relevant information gaps 2 Related Works Scene Representation for Embodied Agents. Constructing an effective scene representation is fundamental for embodied agents, since it determines what information is preserved for reasoning and exploration. Dense 3D

Read the complete text