MomaGraph : State-Aware Unified Scene Graphs with Vision–Language Model for Embodied Task Planning
Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath · 2025
Read MomaGraph : State-Aware Unified Scene Graphs with Vision–Language Model for Embodied Task Planning by Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16909.txt.
Introduction
Yuanchen Ju and coauthors introduce MomaGraph, a scene graph format for household robots that must both move and handle objects. A good scene record must say where things are, what they do, and which parts can be grabbed, and it must update as states change.
MomaGraph merges spatial and functional links with part-level detail in one task-aware graph. The team also built MomaGraph-Scenes, a large annotated data set, and MomaGraph-Bench, a test suite over six reasoning skills. Their 7 billion parameter vision-language model, trained with reward learning, predicts such graphs and plans from them. It hit 71.6 percent on the bench, 11.4 points past the best rival, and transferred to real robots.
Catalog record
| CREATOR | Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MomaGraph : State-Aware Unified Scene Graphs with Vision–Language Model for Embodied Task Planning Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath Abstract Mobile manipulators in households must both navigate and manipulate. This requires a compact, semantically rich scene representation that captures where objects are, how they function, and which parts are actionable. Scene graphs are a natural choice, yet prior work often separates spatial and functional relations, treats scenes as static snapshots without object states or temporal updates, and overlooks information most relevant for accomplishing the current task. To address these limitations, we introduce MomaGraph , a unified scene representation for embodied agents that integrates spatial-functional relationships and part-level interactive elements. However, advancing such a representation requires both suitable data and rigorous evaluation, which have been largely missing. We thus contribute MomaGraph-Scenes , the first large-scale dataset of richly annotated, task-driven scene graphs in household environments, along with MomaGraph-Bench , a systematic evaluation suite spanning six reasoning capabilities from high-level planning to fine-grained scene understanding. Built upon this foundation, we further develop MomaGraph-R1 , a 7B vision–language model trained with reinforcement learning on MomaGraph-Scenes . MomaGraph-R1 predicts task-oriented scene graphs and serves as a zero-shot task planner under a Graph-then-Plan framework. Extensive experiments demonstrate that our model achieves state-of-the-art results among open-source models, reaching 71.6% accuracy on the benchmark ( +11.4% over the best baseline), while generalizing across public benchmarks and transferring effectively to real-robot experiments. † † footnotetext: ∗ Equal Contribution, † Equal Advising Figure 1: Overview of the MomaGraph . Given a task instruction, MomaGraph constructs a task-specific scene graph that highlights relevant objects and parts along with their spatial-functional relationships, enabling the robot to perform spatial understanding and task planning. 1 Introduction When mobile manipulators ( Qiu et al. 2024 ; Honerkamp et al. 2024a ; Wu et al. 2023 ; Zhang et al. 2024a ) enter household environments, they face the fundamental challenge of understanding how the environment works, which objects are interactive, and how they can be used. In other words, such robots must not only be capable of navigating through the home, but also manipulate objects within it. While navigation requires modeling the overall spatial layout, manipulation demands capturing more fine-grained object affordances ( Ju et al. 2024 ; Zhu et al. 2025 ) . This naturally raises a central question: What is the most effective, compact, and semantically rich representation of an indoor scene? An intuitive answer is the scene graph , which organizes objects and their relationships in a scene through a graph structure ( Armeni et al. 2019 ; Koch et al. 2024a ; Koch et al. 2024b ) and has shown great potential in various downstream robotic applications ( Rana et al. 2023 ; Werby et al. 2024 ; Ekpo et al. 2024 ) . However, existing scene graphs suffer from notable limitations. (1) Their edges typically encode only a single type of relationship, either spatial ( Jatavallabhula et al. 2023 ; Gu et al. 2024a ; Loo et al. 2025 ) or functional ( Zhang et al. 2025 ; Dong et al. 2021 ) ( e.g., a remote controlling a TV, a knob adjusting parameters ). Relying solely on spatial relationships captures geometric layout but overlooks operability, while relying solely on functional relationships ignores spatial constraints, leading to incomplete and less actionable structures. (2) Most existing methods ( Wu et al. 2021 ; Takmaz et al. 2025 ; Zhang et al. 2021 ) are limited to static scenes and struggle to adapt to dynamic environments where object positions change or object states change. (3) They lack task relevance, as they fail to emphasize information directly tied to task execution, thereby reducing efficiency and effectiveness. In contrast, cognitive science research ( Uithol et al. 2021 ; Kondyli et al. 2020 ; Castanheira et al. 2025 ) shows that human perception in new environments is both dynamic and task-oriented. Humans do not process all information equally; instead, they flexibly adjust their attention according to the current task. This process is similar to browsing a map on an iPad: people first take a broad view to roughly locate the area of interest, and then zoom in to focus on the specific details needed for the task. Motivated by these insights, we emphasize that an ideal scene graph should integrate both spatial and functional relationships, include fine-grained object parts as nodes, making the representation compact, adaptive to dynamic changes, and highly aligned with task instructions, thus providing a more concrete guidance for embodied perception and task planning. To achieve this goal, we present MomaGraph , a novel scene representation specifically designed for embodied agents. It is the first to unify spatial and functional relationships while introducing part-level interactive nodes, providing a more fine-grained, compact, and task-relevant structured representation than existing approaches. To support this representation, we build MomaGraph-Scenes , the first dataset that jointly models spatial and functional relationships with part-level annotations, encompassing multi-view observations, executed actions, and their interactive object parts, and task-aligned scene graph annotations. Building on this foundation, we propose MomaGraph-R1 , a 7B vision–language model (VLM) trained with the DAPO ( Yu et al. 2025 ) reinforcement learning algorithm on MomaGraph-Scenes . We design a graph-alignment reward function to guide the model toward constructing accurate, task-oriented scene graphs. MomaGraph-R1 not only predicts scene graphs but also serves as a zero-shot task planner within a Graph-then-Plan framework: the model first generates a structured scene graph as an intermediate representation and then performs task planning based on this graph, significantly improving reasoning effectiveness and interpretability. Despite progress in task-graph planning ( Agia et al. 2022 ) , the community still lacks a unified benchmark to systematically evaluate whether and how task-oriented scene graphs improve planning performance. To address this gap, we introduce MomaGraph-Bench , a comprehensive evaluation suite that systematically assesses six key reasoning capabilities, spanning from high-level task planning to fine-grained scene understanding. In summary, our work makes the following key contributions: • We propose MomaGraph , the first scene graph representation that jointly models spatial and functional relationships while incorporating part-level interactive nodes, providing a compact, dynamic, and task-aligned knowledge structure for embodied intelligence. • We construct MomaGraph-Scenes , the first large-scale dataset of richly annotated, task-driven scene graphs in household environments, and build MomaGraph-Bench , a unified evaluation suite that systematically measures the impact of scene graph representations on task planning across six core reasoning capabilities. • We develop MomaGraph-R1 , a 7B vision-language model that leverages reinforcement learning to optimize spatial–functional reasoning, enabling zero-shot planning in a Graph-then-Plan paradigm. • MomaGraph-R1 surpasses all open-source baseline models, delivering substantial gains across public benchmarks and translating these improvements into strong generalization and effectiveness in real-world robotic experiments. 2 Related works Scene Graphs for 3D Indoor Scene Understanding. Scene graphs have emerged as a structured and hierarchical representation in autonomous driving ( Zhang et al. 2024b ; Greve et al. 2024 ) , robot manipulation ( Jiang et al. 2024 ; Wang et al. 2025 ; Engelbracht et al. 2024 ; Jiang et al. 2025 ) , and spatial intelligence ( Yin et al. 2025 ; Zemskova &