Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu · 2025
Read Learning Primitive Embodied World Models: Towards Scalable Robotic Learning by Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20840.txt.
Introduction
Qiao Sun and a large team propose a leaner way in 2025 for robots to learn a model of the world. Video-based world models let robots imagine the results of their actions, but training them needs huge amounts of robot data that are hard to gather. The team noticed that robot motions, though varied, rest on a small set of basic moves.
Their models, called PEWM, generate only short clips of one basic move at a time. Short clips align words with actions more finely and train faster. A vision-language planner then chains the moves into long tasks, guided by start and goal markers. The authors say this split supports flexible control and new combinations of learned moves.
Catalog record
| CREATOR | Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Primitive Embodied World Models: Towards Scalable Robotic Learning Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu Abstract While video-generation-based embodied world models have gained increasing attention, their reliance on large-scale embodied interaction data remains a key bottleneck. The scarcity, difficulty of collection, and high dimensionality of embodied data fundamentally limit the alignment granularity between language and actions and exacerbate the challenge of long-horizon video generation—hindering generative models from achieving a "GPT moment" in the embodied domain. There is a naive observation: the diversity of embodied data far exceeds the relatively small space of possible primitive motions . Based on this insight, we propose a novel paradigm for world modeling– Primitive Embodied World Models (PEWM). By restricting video generation to fixed short horizons, our approach 1) enables fine-grained alignment between linguistic concepts and visual representations of robotic actions, 2) reduces learning complexity, 3) improves data efficiency in embodied data collection, and 4) decreases inference latency. By equipping with a modular Vision-Language Model (VLM) planner and a Start-Goal heatmap Guidance mechanism (SGG), PEWM further enables flexible closed-loop control and supports compositional generalization of primitive-level policies over extended, complex tasks. Our framework leverages the spatiotemporal vision priors in video models and the semantic awareness of VLMs to bridge the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence. 1 Introduction “We can only see a short distance ahead, but we can see plenty there that needs to be done.” —— Alan Turing Embodied agents capable of planning and decision-making in complex environments rely heavily on internal representations of the world, commonly referred to as world models . Recent advances in pretrained video generation models–powered by self-supervised learning on vast internet-scale datasets–have demonstrated remarkable capabilities in synthesizing fine-grained, temporally coherent visual sequences from natural language descriptions [ 27 ] . These successes have inspired a growing body of work to leverage such models as world models for embodied agents, casting policy learning as a video generation problem [ 48 , 27 , 109 , 12 , 6 , 102 , 13 , 38 , 117 , 23 , 57 , 21 , 38 , 54 , 93 , 76 , 115 , 118 , 34 ] : given a language-specified goal, the model generates a future video roll-out, from which actions are subsequently extracted. This paradigm offers compelling advantages in interpretability, generalization, and unified representation across diverse environments. Despite this promise, two fundamental challenges remain largely unaddressed: First, we should rethink alignment before scaling . Current approaches exhibit a trend toward ever-larger models and longer generation horizons, under the assumption that extended prediction leads to better planning . However, we argue that such scaling is fundamentally constrained by the nature of embodied data : the high dimensionality, sparsity, and collection difficulty of real-world interaction data severely limit the feasibility of long-horizon, high-fidelity video prediction. Moreover, fine-grained alignment between linguistic concepts and low-level actions becomes increasingly ill-posed as the prediction window grows. Second, data is the elepant in the room . Training embodied world models critically depends on data–especially in a field where embodied intelligence itself lacks standardization, from sensor configurations to ontologies. Yet, most existing work focuses heavily on model optimization while largely overlooking the design of the underlying data. In practice, the gains from hastily training on a few fragmented, open-source datasets with highly scattered distributions are often limited. A more principled approach–co-designing the data strategy from the ground up in tandem with the model–can yield significantly better results. Figure 1 illustrates this insight through a heuristic analysis. In this work, we challenge the prevailing long-horizon paradigm and propose a shift toward a promising primitive-level modeling . We introduce Primitive Embodied World Models (PEWM) , a new paradigm that restricts video generation to short, fixed-length horizons. By focusing on predicting immediate, primitive-scale transitions, PEWM enables (1) fine-grained alignment between language and action, (2) reduced modeling complexity, (3) improved data efficiency in collection and training, and (4) lower inference latency–effectively unlocking the world model’s potential as a “cerebellum” for fast, reactive control. Figure 1: While densely distributed data can enable broad generalization, embodied data often suffer from sparsity [ 26 , 106 ] . The rightmost schematic highlights how organizing embodied data at the primitive level–along orthogonal dimensions such as action and object–supports compositional generalization even under limited data availability. Our framework bridges the gap between low-level “cerebellum” -like dynamics modeling and high-level “cerebral cortex” -inspired planning, paving the way toward scalable, interpretable, and truly general embodied intelligence. We demonstrate PEWM’s effectiveness in simulation and real-robot experiments, showing strong generalization to novel instructions, robustness to domain shifts, and efficient adaptation with minimal task-specific data. Figure 2: Illustration of primitive-level task execution for “Pick up the yellow tape measure.” 6-DoF motions are meant to be rolled out via diffusion, while discrete gripper actions are handled directly through symbolic execution. Note that this is a simple task, chosen for ease of illustration. 2 Primitive Embodied Data Paving the Way Towards Scalable Embodied Learning 2.1 The Motivation and Definition of Primitive Embodied Data Collecting real-world embodied data is labor-intensive. Existing datasets, despite size, suffer from limited generalization beyond minor scene variations [ 106 , 9 , 22 , 45 , 98 , 23 ] . To improve generalization in embodied agents, we advocate organizing and collecting data at the level of semantically grounded primitives: Definition 2.1 (Primitive as Irreducible Generator) . A primitive is a non-constant, finite-duration trajectory p\in C^{0}([0,T],X) , T>0 , that is indecomposable under concatenation: there do not exist \xi_{1},\xi_{2}\in C^{0}([0,\cdot],X) with 0<\mathrm{dom}(\xi_{1}),\mathrm{dom}(\xi_{2})<T such that p=\xi_{1}\cdot\xi_{2} , and such that p is minimal in the generating set of the behavioral semigroup \mathcal{S}\subset\mathcal{T}(X) . This is grounded in a simple yet powerful observation: the diversity of embodied behaviors vastly exceeds the space of feasible primitive motions. Formally, we posit: Assumption 2.2 (Finite Primitive Basis) . The set of motion primitives \mathcal{P} is finite, and the space of embodied behaviors \mathcal{D}_{\text{emb}} is approximated arbitrarily well by compositions of elements from \mathcal{P} . A full foundation to derive Assumption 2.2 from minimal empirical axioms is presented in Appendix A . This rationale is also illustrated in Figure 1 , which presents a heuristic visualization of the impact of primitive structuring on generalization. Figure 2 illustrates a quick example. Our design advances world model learning in two orthogonal ways : 1) enabling focused, semantically aligned primitive learning and 2) supporting compositional generalization . Short-horizon primitives reduce temporal complexity, helping the model capture key interaction dynamics with greater precision. Their atomic structure allows complex behaviors to be composed from simple elements, enhancing generalizability. Additionally, the semantic granularity simplifies labeling via vision-language models, while short sequences lower computational and optimization burdens in diffusion training. 2.2 Data Collection: Enhancement in Efficiency, Density, and Quality We construct a primitive-centric dataset \mathcal{D}_{\text{prim}}=\{E_{i}\} , where each episode E_{i} is decomposed into primitives \{\mathcal{P}^{k}\}_{k=1}^{N_{i}} with \mathcal{P}^{k}=(\mathbf{x}_{t^{k}}^{\text{img}},\text{Instr}^{k},H^{k}_{s\to g},\mathbf{x}_{t^{k}+1:t^{k+1}-1}^{\text{img}}) . Here, \mathbf{x}_{t^{k}}^{\text{img}} is the initial image, \text{Instr}^{k} the instruction, H^{k}_{s\to g} the start-goal heatmap, and future frames \mathbf{x}_{t^{k}+1:t^{k+1}-1}^{\text{img}} are to be predicted. To enhance data