EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development

Xueyang Zhou, Yihan Sun, Xijie Gong, Guiyao Tie Pan Zhou, Lichao Sun, Yongchao Chen School of Cyber Science and Engineering, Huazhong University of Science and Technology School of Mechanical Science and Engineering, Huazhong University of Science and Technology School of Information and Software Engineering, University of Electronic Science and Technology of China Lehigh University College of AI, Tsinghua University [email protected] , [email protected] , [email protected] · 2026

Read EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development by Xueyang Zhou, Yihan Sun, Xijie Gong, Guiyao Tie Pan Zhou, Lichao Sun, Yongchao Chen School of Cyber Science and Engineering, Huazhong University of Science and Technology School of Mechanical Science and Engineering, Huazhong University of Science and Technology School of Information and Software Engineering, University of Electronic Science and Technology of China Lehigh University College of AI, Tsinghua University [email protected] , [email protected] , [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13800.txt.

Introduction

Zhou and six colleagues turn embodied AI development into conversation. Building test environments, gathering trajectories, training models, and scoring them across tasks and scenes eats engineering time. Their EmbodiedClaw agent lets users state goals and limits in plain conversation, then plans and runs the workflow itself. Environment building, benchmark conversion, trajectory synthesis, model scoring, and asset growth become runnable skills. The shift is from manual toolchains to workflows you can talk through, with work that runs, stays consistent, and repeats well.

Catalog record

CREATORXueyang Zhou, Yihan Sun, Xijie Gong, Guiyao Tie Pan Zhou, Lichao Sun, Yongchao Chen School of Cyber Science and Engineering, Huazhong University of Science and Technology School of Mechanical Science and Engineering, Huazhong University of Science and Technology School of Information and Software Engineering, University of Electronic Science and Technology of China Lehigh University College of AI, Tsinghua University [email protected] , [email protected] , [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32058
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development Xueyang Zhou, Yihan Sun, Xijie Gong, Guiyao Tie Pan Zhou, Lichao Sun, Yongchao Chen School of Cyber Science and Engineering, Huazhong University of Science and Technology School of Mechanical Science and Engineering, Huazhong University of Science and Technology School of Information and Software Engineering, University of Electronic Science and Technology of China Lehigh University College of AI, Tsinghua University [email protected] , [email protected] , [email protected] Abstract Embodied AI research is increasingly moving beyond single-task, single-environment policy learning toward multi-task, multi-scene, and multi-model settings. This shift substantially increases the engineering overhead and development time required for stages such as evaluation environment construction, trajectory collection, model training, and evaluation. To address this challenge, we propose a new paradigm for embodied AI development in which users express goals and constraints through conversation, and the system automatically plans and executes the development workflow. We instantiate this paradigm with EmbodiedClaw, a conversational agent that turns high-frequency, high-cost embodied research activities, including environment creation and revision, benchmark transformation, trajectory synthesis, model evaluation, and asset expansion, into executable skills. Experiments on end-to-end workflow tasks, capability-specific evaluations, human researcher studies, and ablations show that EmbodiedClaw reduces manual engineering effort while improving executability, consistency, and reproducibility. These results suggest a shift from manual toolchains to conversationally executable workflows for embodied AI development. 1 Introduction As embodied AI moves from single-task, single-environment research prototypes toward multi-task, multi-scene, and multi-model development, the bottleneck of embodied research is shifting from solving individual tasks to managing increasingly complex development workflows [ 6 , 23 , 7 , 35 , 22 , 29 ] . Researchers now spend substantial time on environment construction, trajectory collection, model training, and evaluation, while repeatedly adapting tasks, assets, and model backends across platforms. This engineering overhead slows the iteration of ideas and algorithms, increases development cost, and makes embodied research harder to execute, reproduce, and extend [ 2 , 24 , 28 , 12 , 41 ] . Despite rapid progress in simulators, benchmarks, and automated data generation methods [ 15 , 34 , 36 , 45 , 25 , 50 , 37 , 41 , 4 , 47 ] , embodied AI development remains highly platform-bound and difficult to scale. Much of this burden comes from learning platform-specific task definitions and asset organizations, deploying environments, and adapting code across simulators, benchmarks, and model stacks [ 16 , 9 , 30 , 3 , 26 , 11 ] . In practice, these efforts repeatedly concentrate in some high-frequency activities: simulation task creation, task revision, trajectory collection, model training and evaluation [ 14 , 17 , 37 ] . Inspired by "vibe coding", we argue that embodied AI development should move from manually assembling platform-specific experimental loops to conversationally driven execution of embodied development skills. Therefore, we present EmbodiedClaw, which turns high-frequency, time-consuming embodied engineering tasks into an executable workflow driven by user intent through dialogue. It organizes embodied development around three classes of objects: simulation environments, data, and models. On top of these objects, EmbodiedClaw supports three core capabilities: simulation environment synthesis, trajectory data collection, and model deployment. This formulation is motivated by recent progress in language-driven embodied agents and general-purpose workflow automation systems [ 1 , 31 , 10 , 32 ] , while targeting the more structured and tool-dependent setting of embodied AI development [ 13 , 19 , 39 , 5 , 51 ] . We evaluate this paradigm through end-to-end workflow tasks, capability-specific evaluations, human researcher studies, and ablations. Our results show that EmbodiedClaw reduces manual engineering burden while improving executability, consistency, and reproducibility in embodied AI development. Our main contributions are as follows: • We propose a new view of embodied AI research in which high-frequency and time-intensive engineering activities are unified as a class of conversationally driven executable development skills. This abstraction turns previously fragmented engineering routines into automatic workflows and compresses workflows that previously required days of manual effort into only a few hours. • We introduce EmbodiedClaw, a dialogue-driven agent system that translates user input into executable workflows and supports arbitrary operations over three core embodied research objects: simulation environments, data, and models. • Extensive experiments show that EmbodiedClaw delivers substantial gains on a range of high-cost embodied development tasks, including batch simulation environment creation and editing, trajectory data collection, and model deployment and evaluation. In these settings, EmbodiedClaw achieves up to an order-of-magnitude improvement in efficiency while reaching accuracy close to that of human experts. 2 Preliminary Figure 1: Overview of EmbodiedClaw capabilities. The framework supports batch simulation environment synthesis, including automatic scene creation and controllable scene editing, batch trajectory collection for embodied manipulation tasks, and adaptive export of embodied data to HDF5, LeRobot, RLDS, and video formats. It further enables model deployment with vision-language and world models, model training through imitation learning and reinforcement learning, and model evaluation on standard benchmarks such as LIBERO [ 21 ] , RoboTwin [ 28 ] , and SimplerEnv [ 18 ] . 2.1 EmbodiedClaw We formulate EmbodiedClaw as a conversational framework that maps user interaction into executable embodied development workflows over three operational objects: executable environments o_{t}^{S} , data o_{t}^{D} , and model/evaluation states o_{t}^{M} . At step t , the operational context is \mathbf{o}_{t}=(o_{t}^{S},o_{t}^{D},o_{t}^{M})\in\mathcal{O}^{S}\times\mathcal{O}^{D}\times\mathcal{O}^{M}. (1) Given dialogue context c_{t} and user input x_{t} , EmbodiedClaw infers an intent representation z_{t}^{I}=\psi(c_{t},x_{t}) and uses it to synthesize an executable workflow \pi_{t}=\phi(z_{t}^{I},\mathbf{o}_{t}) , where \pi_{t}\in\Pi is a sequence of skill calls. Conditioned on z_{t}^{I} , the workflow updates the operational objects according to (o_{t}^{S},o_{t}^{D},o_{t}^{M})\xrightarrow{\pi_{t}\mid z_{t}^{I}}(o_{t+1}^{S},o_{t+1}^{D},o_{t+1}^{M}). (2) The objective is to minimize development cost while producing outcomes aligned with the inferred intent. We therefore define \pi_{t}^{*}=\arg\min_{\pi\in\Pi}\underbrace{\mathcal{C}_{\mathrm{human}}(\pi)+\alpha\,\mathcal{C}_{\mathrm{sys}}(\pi)}_{\text{Cost / Overhead}}+\underbrace{\lambda\,d\!\left((o_{t+1}^{S},o_{t+1}^{D},o_{t+1}^{M}),\mathbf{o}^{*}_{\mathrm{op}}(z_{t}^{I})\right)}_{\text{Goal Deviation}}, (3) where \mathcal{C}_{\mathrm{human}}(\pi) denotes the required human effort, \mathcal{C}_{\mathrm{sys}}(\pi) denotes system execution cost, including time and token usage, and \mathbf{o}^{*}_{\mathrm{op}}(z_{t}^{I}) denotes the target operational outcome implied by the inferred intent representation. 2.2 Embodied AI Research Operations We identify three major classes of high-frequency and time-intensive operations in embodied AI research: simulation environment synthesis, trajectory synthesis, and model deployment. Given dialogue context c_{t} , user input x_{t} , and an optional initial simulator state o_{t}^{S,\mathrm{sim}} , the system infers an intent representation z_{t}^{I}=\psi(c_{t},x_{t}) and synthesizes a workflow that either creates a new simulator state or edits an existing one. Formally, \displaystyle\pi_{t}^{\mathrm{env}} \displaystyle=\phi(z_{t}^{I},o_{t}^{S}), (4) \displaystyle\tilde{o}_{t}^{S} \displaystyle\xrightarrow{\pi_{t}^{\mathrm{env}}}o_{t+1}^{S}, \displaystyle d\!\left(o_{t+1}^{S},\mathbf{o}^{*}_{\mathrm{op}}(z_{t}^{I})\right) \displaystyle=\Delta_{\mathrm{goal}}+\rho\,\Delta_{\mathrm{pres}}, where \tilde{o}_{t}^{S} denotes either an initialization state or an editable simulator state, \Delta_{\mathrm{goal}} measures consistency with the explicit requirements specified in the user intent, including objects, spatial relations, robot or manipulator configurations, and other environment-level constraints, and \Delta_{\mathrm{pres}} measures unintended changes to content that should remain unchanged during controllable editing. Trajectory Synthesis. EmbodiedClaw models trajectory synthesis as the generation of training-ready data from executable simulator states. This class includes both training trajectory collection and training data transformation, such as converting raw rollouts into model-consumable supervision formats. Given a simulator state o_{t}^{S} and task specification x_{t} , the system executes a data-generation workflow to produce processed trajectory data o_{t}^{D} . Formally, \displaystyle\pi_{t}^{\mathrm{traj}} \displaystyle=\phi(o_{t}^{S},x_{t}), (5) \displaystyle(o_{t}^{S},x_{t}) \displaystyle\xrightarrow{\pi_{t}^{\mathrm{traj}}}o_{t}^{D}, \displaystyle d\!\left(o_{t}^{D},\mathbf{o}^{*}_{\mathrm{op}}(x_{t},o_{t}^{S})\right) \displaystyle=\Delta_{\mathrm{task}}+\eta\,\Delta_{\mathrm{stab}}+\kappa\,\Delta_{\mathrm{fmt}}, where o_{t}^{D} may contain collected trajectories, relabeled transitions, and transformed training samples, \Delta_{\mathrm{task}} measures how well the synthesized data supports the target task, \Delta_{\mathrm{stab}} measures consistency and robustness across repeated rollouts or processing

Read the complete text