A Step Toward World Models: A Survey on Robotic Manipulation
Peng-Fei Zhang, Ying Cheng, Xiaofan Sun, Shijie Wang, Fengling Li, Lei Zhu, Heng Tao Shen · 2025
Read A Step Toward World Models: A Survey on Robotic Manipulation by Peng-Fei Zhang, Ying Cheng, Xiaofan Sun, Shijie Wang, Fengling Li, Lei Zhu, Heng Tao Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02097.txt.
Introduction
Peng-Fei Zhang and six co-authors survey how robots that manipulate objects are beginning to use world models, internal representations that capture how an environment behaves. Robots expected to act in messy, changing settings need more than reactive control; they need to predict, plan, and reason. Yet the field still lacks agreement on what a world model is, what it covers, and what it must be able to do.
Rather than imposing one definition, the survey examines methods from robotic manipulation that display the core abilities of world models, whether or not they carry the label. It organises their roles in perception, prediction, and control, catalogues recurring challenges and fixes, and distils the components a fully realised world model for robotics should have.
Catalog record
| CREATOR | Peng-Fei Zhang, Ying Cheng, Xiaofan Sun, Shijie Wang, Fengling Li, Lei Zhu, Heng Tao Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Step Toward World Models: A Survey on Robotic Manipulation Peng-Fei Zhang, Ying Cheng, Xiaofan Sun, Shijie Wang, Fengling Li, Lei Zhu, Heng Tao Shen Abstract Autonomous agents are increasingly expected to operate in complex, dynamic, and uncertain environments, performing tasks such as manipulation, navigation, and decision-making. Achieving these capabilities requires agents to understand the underlying mechanisms and dynamics of the world, moving beyond reactive control or simple replication of observed states. This motivates the development of world models as internal representations that encode environmental states, capture dynamics, and support prediction, planning, and reasoning. Despite growing interest, the definition, scope, architectures, and essential capabilities of world models remain ambiguous. In this survey, we go beyond prescribing a fixed definition and limiting our scope to methods explicitly labeled as world models. Instead, we examine approaches that exhibit the core capabilities of world models through a review of methods in robotic manipulation. We analyze their roles across perception, prediction, and control, identify key challenges and solutions, and distill the core components, capabilities, and functions that a fully realized world model should possess. Building on this analysis, we aim to motivate further development toward generalizable and practical world models for robotics. This is an initial version of the survey. The content will be expanded and refined in future updates. I Introduction “ If I have seen further, it is by standing on the shoulders of giants .” — Isaac Newton Fig. 1: Conceptual flow of the survey. The survey aims to clarify the motivation behind world modeling, explore its essential scope and development, and illuminate pathways toward more general and capable world models. Autonomous agents are designed to extend human capabilities, assisting in tasks that are dangerous, repetitive, or demand high precision, thereby enhancing productivity across diverse applications. Robots serve as their physically embodied form, aiming to realize seamless interaction with the physical world. Achieving such capabilities requires agents to move beyond reactive control and the mere imitation of observed behaviors, instead developing the ability to model, reason about, and predict environmental dynamics. In this context, world models have emerged as powerful internal representations that enable robots to anticipate future outcomes, plan effectively, and ultimately act intelligently in the real world. Richens et al. [ 137 ] argue that any agent capable of generalizing to solve multi-step tasks must implicitly learn a predictive model of its environment, e.g., a world model. The concept of “world models” in computer science dates back to the 1960s [ 122 ] , and numerous methods have since been proposed toward more capable models [ 142 , 67 , 45 , 176 , 188 , 84 , 166 , 182 ] , though not all explicitly identify as world models. For example, Wang et al. [ 166 , 182 , 45 , 176 , 188 , 84 ] leverage video generation models as a form of world models, which encode extensive world knowledge from large-scale training data and can predict future states based current observations and/or actions. LeCun et al. [ 103 , 68 , 70 , 177 , 71 ] emphasizes modeling abstract world state representations, while Zitkovich et al. [ 207 , 83 , 81 ] utilize vision-language-action models (VLA) models that do not explicitly generate future states. The scope of existing methods varies from 2D scene prediction to 4D world modeling [ 183 , 20 , 64 , 22 , 205 , 196 ] , reflecting different understandings of what it means to model the world. The observation viewpoint of the world includes both third-person (exocentric) [ 64 , 53 , 163 ] and first-person (egocentric) [ 29 , 61 ] perspectives. World models play a critical role in robotic learning in two ways. They allow robots to improve autonomous policies by the internal prediction and simulation of environmental dynamics, future states, and/or action outcomes [ 69 , 68 , 70 , 177 , 48 , 64 ] . They also support scalable policy training and evaluation by generating realistic rollouts and physical interactions [ 207 , 16 , 156 , 12 , 114 , 112 ] . From a functional standpoint, current approaches range from single-purpose models, such as those for visual planning [ 48 , 54 ] , future-scene generation [ 152 , 12 , 87 ] , or action prediction [ 145 ] , to more integrated systems that couple multiple abilities within a unified framework [ 24 , 29 , 195 , 149 , 204 ] . These variations indicate that the notion of a world model remains unsettled, with its conceptual, architectural, and functional boundaries not yet clearly defined. Addressing these questions requires standing on the shoulders of prior contributions, carefully analyzing existing methodologies to gain inspiration for elucidating the boundaries of world models. In this survey, rather than hastily defining what constitutes a world model, we provide a comprehensive review of the literature, highlighting their core principles, architectures, and functional roles in enabling intelligent robotic systems. We extend the scope beyond works explicitly labeled as world models, examining their core principles and outlining pathways for constructing practical models that can drive the development of general and adaptive robotic agents. This survey is organized around a set of guiding questions designed to provoke thought and provide inspiration. Readers can explore the survey with these questions in mind, using them to provoke thought, gain inspiration, and reflect on the challenges and opportunities in developing world models for robotic manipulation. • What is the world model and its conceptual, architectural, and functional boundaries? • How should the world be sensed and presented? • What level of model fidelity and coverage is required to reliably support robotic tasks? • Is it necessary to learn a world model, given the complexity, resource demands, and potential challenges involved? • How far are current world models from fully realized world models? • Is human cognition [ 14 , 167 ] the ultimate goal for world models? The main contributions of this survey are as follows: • Comprehensive taxonomy of world model architectures. We provide a systematic categorization of existing designs, including latent space modeling methods, video generation-based models, direct projection based (implicit modeling) methods and other emerging structures. • Functional analysis. We discuss diverse roles of world models in robotic manipulation, including action prediction and planning, robotic learning, and evaluation, highlighting their contribution to intelligent manipulation. • Capability framework. We analyze the essential abilities that a world model should possess, such as perception, prediction, imagination, and interaction, aiming to clarify what constitutes a generalizable and capable world model. • Challenges and future directions. We summarize key challenges, including long-horizon reasoning, physical awareness, and generalization, and discuss potential research directions and solutions toward building practical, real-world models. Related Surveys. Our survey differs substantially from existing reviews. Several prior surveys have examined world models in robotics, but most focus on specific aspects and provide limited conceptual analysis. For example, Yu et al. [ 187 ] emphasize video generation, Kong et al. [ 102 ] cover 3D/4D world modeling, Ai et al. [ 5 ] study dynamics learning, and Lin et al. [ 110 ] address physics cognition. Long et al. [ 113 ] review architectures and functional roles