Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
Sen Cui, Jingheng Ma · 2026
Read Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling by Sen Cui, Jingheng Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00412.txt.
Introduction
Sen Cui and Jingheng Ma argue, from a Hamiltonian angle, that world models for robots and cars should be built around physics. World models, which predict how a scene will unfold, have split into three camps: video makers, 3D scene builders, and latent forecast models. Each camp, they note, still struggles to give reliable forecasts that actions can steer and that stay stable over long spans.
Their plan encodes what the sensors see into an ordered latent phase space, the state language of Hamiltonian mechanics. The latent state then evolves by Hamiltonian-inspired rules with terms for control, losses, and leftovers, before decoding into forecasts for planning. The authors discuss possible gains in clarity, data thrift, and steadiness. They also flag hard cases plainly: friction, contact, energy-losing forces, and soft objects all strain the approach.
Catalog record
| CREATOR | Sen Cui, Jingheng Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00412
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling Sen Cui, Jingheng Ma Abstract World models have recently re-emerged as a central paradigm for embodied intelligence, robotics, autonomous driving, and model-based reinforcement learning. However, current world model research is often dominated by three partially separated routes: 2D video-generative models that emphasize visual future synthesis, 3D scene-centric models that emphasize spatial reconstruction, and JEPA-like latent models that emphasize abstract predictive representations. While each route has made important progress, they still struggle to provide physically reliable, action-controllable, and long-horizon stable predictions for embodied decision making. In this paper, we argue that the bottleneck of world models is no longer only whether they can generate realistic futures, but whether those futures are physically meaningful and useful for action. We propose Hamiltonian World Models as a physically grounded perspective on world modeling. The key idea is to encode observations into a structured latent phase space, evolve the latent state through Hamiltonian-inspired dynamics with control, dissipation, and residual terms, decode the predicted trajectory into future observations, and use the resulting rollouts for planning. We discuss how Hamiltonian structure may improve interpretability, data efficiency, and long-horizon stability, while also noting practical challenges in real-world robotic scenes involving friction, contact, non-conservative forces, and deformable objects. 1 Introduction The idea of learning internal simulators has regained prominence as a unifying objective across embodied AI, robotics, and decision making. This renewed interest is driven by two converging trends. First, large-scale foundation models have demonstrated strong semantic competence in language, vision, and multimodal reasoning ( 28 ; 1 ; 27 ; 10 ; 6 ) . Second, embodied agents require predictive models that go beyond recognizing the current scene: they must anticipate how the world will change under their own actions. A robot, for example, should not only identify an object, but also predict whether pushing, grasping, or releasing it will lead to stable manipulation, collision, or task failure. This makes world modeling a natural bridge between perception, prediction, and decision making. The classical motivation for world models is to provide an internal simulator of the environment, enabling agents to imagine possible futures before acting. Early work showed that learned latent models can support control by compressing observations into compact predictive representations ( 16 ) . Model-based reinforcement learning further demonstrated that agents can plan or optimize policies inside learned latent dynamics, reducing the need for costly real environment interaction ( 18 ; 17 ; 19 ) . In robotics, visual foresight and video prediction have been used to evaluate candidate actions before execution ( 12 ; 11 ) . In autonomous driving and interactive simulation, generative world models such as GAIA-1 and Genie suggest that future scene generation can be conditioned on actions, text, or latent controls ( 22 ; 7 ) . Together, these developments have made “generating the future” an increasingly important research direction. However, generating future video is not equivalent to understanding the physical world. Modern video generative models can synthesize visually plausible sequences, but visual plausibility alone does not guarantee physical validity. A generated trajectory may look realistic over a short horizon while gradually violating object permanence, contact consistency, momentum conservation, or action-effect causality. This distinction is especially important for robotics and autonomous systems, where an incorrect future prediction can directly mislead planning and execution. For entertainment-oriented video synthesis, perceptual realism may be sufficient; for embodied decision making, the predicted future must also be dynamically feasible, controllable, and reliable. A key limitation of many current video world models is that their temporal evolution is learned through highly expressive but weakly constrained neural sequence models. In a typical latent formulation, the future state is predicted as z_{t+1}=f_{\theta}(z_{t},a_{t}), (1) where z_{t} is a learned latent state and a_{t} is an action. Although this formulation is flexible and compatible with recurrent latent dynamics, diffusion-based video predictors, Transformers, and structured state-space models ( 16 ; 18 ; 21 ; 5 ; 35 ; 15 ; 14 ) , it does not specify what physical quantities the latent state represents or what structure the transition should preserve. As a result, the model may learn short-term correlations in pixel or latent space without acquiring stable physical dynamics. This often leads to compounding errors, long-horizon drift, poor out-of-distribution generalization, and high data requirements. This limitation suggests that the bottleneck of world models is no longer only visual fidelity. The more fundamental question is whether the generated future is physically coherent and useful for action. A world model should not merely predict what the next frames look like; it should represent how objects, agents, and their interactions evolve under interventions. Prior work on model-based control and robotic video prediction has already shown the practical value of imagined rollouts for planning ( 12 ; 11 ; 17 ; 19 ) . Yet for such rollouts to support reliable decision making, the latent dynamics must be stable, interpretable, and constrained by the regularities of the physical world. We therefore argue that the next stage of world model research should move from visual generation toward physical structure modeling. In this view, video generation is an observation-level interface, while the core of a world model is its latent dynamics. The internal state should encode entities, relations, controllable variables, and physically meaningful structure; the transition should reflect constraints such as geometry, energy, contact, and action-conditioned causality; and the generated future should be treated as a rendering of an underlying state evolution. This perspective is consistent with broader efforts in structured dynamics learning, object-centric representation learning, and physics-informed machine learning ( 4 ; 13 ; 34 ; 29 ; 30 ; 25 ) . Motivated by this perspective, we propose to rethink world models through the lens of Hamiltonian dynamics. Hamiltonian mechanics provides a principled framework for describing physical evolution in phase space through generalized coordinates, momenta, and an energy function. Rather than asking a video model to discover physical laws purely from data, a Hamiltonian world model leverages energy-based dynamics in a learned phase space to govern latent evolution. This offers a path toward world models that are not only generative, but also physically interpretable, data-efficient, and stable under long-horizon rollout. 2 Existing Three Routes Toward World Models World model research has recently developed along several partially overlapping but conceptually distinct routes. In this paper, we group current approaches into three representative paradigms: 2D video-generative world models , 3D scene-centric world models , and JEPA-like latent world models . This classification is not meant to be exhaustive, but it clarifies what different routes primarily optimize for. The 2D video route emphasizes future observation synthesis, the 3D route emphasizes spatial and geometric reconstruction, and the JEPA-like route emphasizes predictive abstraction in latent representation space. 2D video-generative world models. The first route treats world modeling as conditional future video generation. Given past observations, actions, language, maps, or other conditioning signals, the model predicts future visual observations: p_{\theta}(o_{t+1:t+H}\mid o_{\leq t},a_{t:t+H-1},c) , where o_{t} denotes an observation, a_{t} denotes an action or control input, and c denotes additional context. This route is closely related to recent advances in video diffusion and latent video generation, which have significantly improved the quality and temporal coherence of generated videos ( 21 ; 5