E m m a - X : An E mbodied M ulti m odal A ction Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

Qi Sun *, Pengfei Hong *, Tej Deep Pala, Vernon Y.H. Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria · 2024

Read E m m a - X : An E mbodied M ulti m odal A ction Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning by Qi Sun *, Pengfei Hong *, Tej Deep Pala, Vernon Y.H. Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11974.txt.

Introduction

Qi Sun and co-authors present Emma-X, a 2024 model that helps robots act on what they see and hear. Classic robot control is built for one task and fails on new objects or orders. Vision-language models read scenes well but cannot turn plans into moves for a given robot body. Emma-X aims to bridge that gap.

The model reasons step by step with its steps tied to real objects in view. It also looks ahead in space before acting. Training used a large set of robot arm paths, labeled with task reasoning and spatial hints. A splitting method based on gripper state cut false grounding. Tests showed Emma-X beating rival systems, above all on real tasks needing spatial care.

Catalog record

CREATORQi Sun *, Pengfei Hong *, Tej Deep Pala, Vernon Y.H. Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

E m m a - X : An E mbodied M ulti m odal A ction Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning Qi Sun *, Pengfei Hong *, Tej Deep Pala, Vernon Y.H. Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria Abstract Traditional reinforcement learning-based robotic control methods are often task-specific and fail to generalize across diverse environments or unseen objects and instructions. Visual Language Models (VLMs) demonstrate strong scene understanding and planning capabilities but lack the ability to generate actionable policies tailored to specific robotic embodiments. To address this, Visual-Language-Action (VLA) models have emerged, yet they face challenges in long-horizon spatial reasoning and grounded task planning. In this work, we propose the Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning, E m m a - X . E m m a - X leverages our constructed hierarchical embodiment dataset based on BridgeV2, containing 60,000 robot manipulation trajectories auto-annotated with grounded task reasoning and spatial guidance. Additionally, we introduce a trajectory segmentation strategy based on gripper states and motion trajectories, which can help mitigate hallucination in grounding subtask reasoning generation. Experimental results demonstrate that E m m a - X achieves superior performance over competitive baselines, particularly in real-world robotic tasks requiring spatial reasoning. We make our codes, models and datasets publicly available: https://declare-lab.github.io/Emma-X/ . 1 Introduction The robotic policy model aims to generate sequences of low-level action manipulation policies for robots. Traditional reinforcement learning-based robotic control methods often focus on narrowly defined tasks within fixed environments ( Ma et al., 2024 ) , hindering their ability to generalize beyond task-specific training data and limiting their applicability ( Brohan et al., 2023b ; Chi et al., ) . Recent advancements in foundation models for vision and language have highlighted the remarkable scene-understanding and task-planning capabilities ( Radford et al., 2021 ; Zhai et al., 2023 ; Touvron et al., 2023 ) . These Visual-Language Models (VLMs) excel at breaking down complex tasks into manageable steps through chain-of-thought reasoning and demonstrate significant potential in planning. Despite their strengths, VLMs are not inherently designed to directly generate policies applicable to specific embodiment configurations in robotics. This limitation has spurred the emergence of Visual-Language-Action (VLA) models, which aim to bridge this gap by leveraging multimodal inputs to produce adaptive and generalized robotic actions for complex, multi-task scenarios ( Brohan et al., 2023a ; Kim et al., 2024 ; Octo Model Team et al., 2024 ) . Figure 1: Comparison of our E m m a - X with ECoT in task reasoning. While both approaches utilize Gemini, our method also incorporates image sequence input, whereas ECoT relies solely on text input. We also illustrate an example of spatial reasoning. However, most of the existing VLA models often exhibit “muscle memory” response patterns, struggling to perceive scene variation and understand instructions as humans do when handling complex tasks or ambiguous commands. Zawalski et al. (2024) attempts to address this issue through visual and task reasoning, including the bounding box of the object, task segmentation, and the direction of predicted action, etc. Although they equip VLAs with an understanding of the current situation and task, they lack long-horizon spatial reasoning on how robots should move next. We hypothesize that the completion of subgoals or subtasks can be enhanced if the VLA incorporates look-ahead spatial reasoning, such as inferring the gripper’s future 2D position and the 3D movement plans necessary for the gripper to reach that position. In particular, we train the VLA model to predict future position g_{t+k} of the gripper as checkpoints and use them to devise a high-level movement plan \beta(g_{t},g_{t+k}) . This plan informs the immediate action a_{t} at the current state s_{t} , ensuring decisions are both reactive to the present and aligned with long-term objectives. Similar to a delivery driver planning a route with key landmarks to make purposeful driving decisions, this approach optimizes task completion by balancing foresight and adaptability. Additionally, another limitation in task reasoning provided by ECoT ( Zawalski et al., 2024 ) is the absence of visual grounding when augmenting reasoning data using Gemini. We observe that Gemini frequently hallucinates due to a lack of holistic understanding of the setup and environment. As shown in Figure 1 , the image shows that the robot already started to grasp the pot cover, while the task reasoning indicates the subtask is still “Move to silver pot cover", which conflicts with the following reasoning they provided. In this work, we introduce the Embodied Multimodal Action Model with Grounded Chain of Thought Reasoning, E m m a - X . We develop a hierarchical embodiment dataset based on BridgeV2, consisting of 60,000 robot manipulation trajectories. For each state of a given trajectory, we generate detailed spatial reasoning grounded in the environment and task reasoning, such as the plans of how the robot should perform the subtask. As shown in Figure 1 , we also generate the 2D gripper position, and 3D spatial movements of the gripper to transit to future states, which enable the VLA model to reason a long-horizon plan for accomplishing the task. Furthermore, we utilize Gemini ( Team et al., 2023 ) to generate grounded task reasoning for each observed state. To avoid the abovementioned reasoning conflict problem of task reasoning in ECoT, we propose a novel trajectory segmentation strategy, which leverages the opening and closing states of the gripper and the motion trajectory of the robot arm to segment the sequence of states into distinct segments. By grounding, we mean that, unlike ECoT, which prompts Gemini to generate subtask reasoning based solely on textual descriptions, our approach incorporates visual images segmented using the aforementioned strategy. As shown in Figure 1 , our method can accurately provide the subtask “Grasping the pot cover" corresponding to the current robotic state. This illustrates that our strategy significantly reduces Gemini’s hallucination issues by requiring it to construct a visual understanding of the environment, rather than relying solely on textual descriptions of the environment. Finally, we train our E m m a - X based on OpenVLA using our constructed hierarchical embodiment dataset. The main contributions of our work are summarized as follows: • We introduce a 7B-parameter embodied multimodal action model, E m m a - X created by fine-tuning OpenVLA with the grounded chain of thought (CoT) reasoning data. • We synthetically construct a hierarchical embodiment dataset from the existing robot manipulation dataset, which includes the 3D spatial movements, 2D gripper position, and grounded reasoning. • We propose a novel trajectory segmentation strategy that leverages the gripper’s opening and closing states alongside the motion trajectory of the robot arm, facilitating both grounded task reasoning and look-ahead spatial reasoning. • Our proposed E m m a - X achieves significant performance improvements over existing competitive baselines on various real-world robot tasks, especially in tasks where spatial reasoning is required. 2 Problem Formulation 2.1 Policy Imitation Learning Given a set of expert demonstrations \mathcal{D}=\{(\{s_{t}\}_{t=1}^{T},\mathcal{T}_{i},\{a_{t}\}_{t=1}^{T})\}_{i=1}^{N} , where N is the number of demonstrations in the dataset, T is the number of states (image frames of the environment) for a data sample D_{i} , s_{i}=\text{image}_{i} represents the state consisting of an image of the environment,

Read the complete text