Robot Learning from a Physical World Model

Jiageng Mao, Sicheng He, Hao-Ning Wu, Yang You, Shuyang Sun, Zhicheng Wang, Yanan Bao, Huizhong Chen, Leonidas Guibas, Vitor Guizilini, Howard Zhou, Yue Wang, [0.5em] Google DeepMind · 2025

Read Robot Learning from a Physical World Model by Jiageng Mao, Sicheng He, Hao-Ning Wu, Yang You, Shuyang Sun, Zhicheng Wang, Yanan Bao, Huizhong Chen, Leonidas Guibas, Vitor Guizilini, Howard Zhou, Yue Wang, [0.5em] Google DeepMind (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07416.txt.

Introduction

Generated videos look convincing, yet copying their pixel motion can teach robots the wrong moves. Jiageng Mao and colleagues introduce PhysWorld to fix that gap. Their framework pairs video generation with a rebuilt model of the physical scene behind the video.

Starting from one image and a task command, the system generates a demonstration video and reconstructs the objects and surfaces in it. Motions are then grounded through object-focused reinforcement learning inside that physical model. Tests suggest the grounded actions transfer to real robot tasks more reliably. The work links generative video to physics-aware robot training.

Catalog record

CREATORJiageng Mao, Sicheng He, Hao-Ning Wu, Yang You, Shuyang Sun, Zhicheng Wang, Yanan Bao, Huizhong Chen, Leonidas Guibas, Vitor Guizilini, Howard Zhou, Yue Wang, [0.5em] Google DeepMind
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robot Learning from a Physical World Model Jiageng Mao, Sicheng He, Hao-Ning Wu, Yang You, Shuyang Sun, Zhicheng Wang, Yanan Bao, Huizhong Chen, Leonidas Guibas, Vitor Guizilini, Howard Zhou, Yue Wang, [0.5em] Google DeepMind Abstract We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit the project webpage for details. I Introduction Recent advances in generative models enable the synthesis of photorealistic videos directly from images and language instructions. Trained on large-scale Internet data, video generation models exhibit strong generalization across diverse scenarios. For robotics, such models offer a powerful source of visual guidance for manipulation. Given a robot’s observation and a task instruction, a video generator can produce a demonstration that depicts task completion. These generated videos inherently capture object dynamics and embodiment motions, which can be leveraged to learn generalizable robotic manipulation policies. Despite the great promise of video generation, translating generated pixel motions into executable robotic actions remains highly challenging. Previous works [ 33 , 52 , 49 , 22 , 12 ] learn inverse dynamics or policy models to align generated video frames with real robotic actions. Yet, such methods generally rely on large-scale real-world demonstrations for alignment, while collecting them at scale is costly and labor-intensive. Other methods [ 27 , 5 , 39 ] propose to extract robotic actions by directly following visual cues, e.g. flows, sparse tracks, or object poses, from generated videos. Nevertheless, directly retargeting video motions to robots neglects underlying physical constraints, often leading to inaccurate manipulations. We argue that the key bottleneck of bridging generated videos and robotic actions lies in physical feasibility. Video generation, despite its generalization power, only provides visual plausibility rather than physical accuracy for robotic tasks, whereas robots operating in the real world require physically accurate actions to interact with objects correctly. We tackle this dilemma by introducing a proxy physical world model built from generated videos. This world model provides realistic physical feedback, enabling scalable robot learning to imitate generated video motions in a physically consistent manner. To this end, we propose PhysWorld, a framework for physically grounded robot learning from video generation. The core of PhysWorld lies in the synergy between physical world reconstruction and video generation: video generation provides pixel-level visual guidance for task execution, while the physical world model offers realistic feedback for learning from the generated visual guidance. In particular, given a single RGB-D image and a task prompt, our method first generates a task-conditioned video depicting how the task is completed visually. Next, we propose a novel method for constructing a physically interactable scene from the video. Finally, we introduce an object-centric residual reinforcement learning approach that bridges video generation and physical world reconstruction, producing physically accurate robotic actions. Our framework requires only a single RGB-D image and a language command, yet outputs executable actions that follow the instruction to complete the task. By explicitly modeling physics, PhysWorld eliminates the need for real-world data collection and achieves zero-shot generalizable robotic manipulation, while significantly improving manipulation accuracy over previous methods. We evaluate PhysWorld on a diverse set of real-world robotic manipulation tasks. Experimental results show that combining video generation with physical world modeling yields substantial improvements in accuracy across all tasks. PhysWorld enables physically grounded and generalizable robotic manipulation, consistently outperforming existing approaches by a large margin. We will release code and project resources to facilitate further research. II Related Works Video generation for robotics. Video generation [ 64 , 2 , 69 ] holds great promise for robotics. It has been explored for goal generation [ 46 ] , planning [ 52 , 11 ] , dynamics learning [ 63 , 29 ] , and policy learning [ 27 , 22 , 33 , 39 , 49 , 5 , 12 ] . To extract robotic actions from generated videos, several works [ 33 , 52 , 49 , 22 , 12 , 63 , 29 ] train action models from generated video frames using large amounts of real robotic demonstrations, but collecting such data is costly. In contrast, PhysWorld removes the need for real-world data collection and enables zero-shot robotic manipulation. Other approaches [ 27 , 5 , 39 ] directly extract actions by following visual cues from generated videos, such as optical flows [ 27 ] , sparse tracks [ 5 ] , or object poses [ 39 ] . However, pixel-level imitation neglects physical plausibility and often results in inaccurate real-world manipulation. PhysWorld instead introduces a proxy physical world model, allowing agents to imitate generated video motions with physical feedback, thereby improving the accuracy and feasibility of real-world robotic manipulation. Robot learning from videos. Videos contain rich motion and task information that can be leveraged for training robotic policies. Researchers tackle this problem by learning transferable representations [ 21 , 4 , 44 , 53 , 23 , 37 , 8 , 45 , 26 , 20 , 61 ] , tracking embodiment-agnostic motion representations [ 58 , 17 , 62 , 18 , 60 , 42 , 28 , 38 , 25 , 19 , 7 , 71 , 55 , 6 ] , real-to-sim [ 36 , 54 , 65 , 10 ] , or reinforcement learning [ 47 , 35 ] . PhysWorld shares similar insights with other works on object pose tracking [ 19 , 39 ] , real-to-sim reconstruction [ 65 , 10 , 35 , 3 ] , and reinforcement learning [ 65 , 35 ] . However, these methods generally rely on ad-hoc laboratory settings for real-to-sim reconstruction and human demonstration collection, which limits their generalization to in-the-wild generated videos that often contain motion blur or visual hallucinations. In contrast, PhysWorld only requires a single generated video for real-to-sim reconstruction and can effectively learn physically accurate robotic actions from generated videos. Fig. 2: PhysWorld pipeline. Given an RGB-D image and a task prompt, our framework (i) generates a task-conditioned video, (ii) reconstructs a geometry-aligned 4D representation from the generated video, (iii) generates textured object and background meshes, (iv) assembles them into a physically interactable scene through property estimation, gravity alignment, and collision optimization, (v) learns object-centric residual RL policies that transform visual demonstrations into feasible robotic actions, and (vi) deploys to the real world. Real-to-sim-to-real. Real-to-sim-to-real methods reconstruct a physical scene from observations and embed it in simulators for policy learning. To obtain complete object

Read the complete text