Video Generators are Robot Policies
Junbang Liang Pavel Tokmakov Ruoshi Liu Sruthi Sudhakar, Paarth Shah Rares Ambrus Carl Vondrick · 2025
Read Video Generators are Robot Policies by Junbang Liang Pavel Tokmakov Ruoshi Liu Sruthi Sudhakar, Paarth Shah Rares Ambrus Carl Vondrick (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00795.txt.
Introduction
Robot control policies trained on human demonstrations falter when objects, rooms, or tasks change, and demonstrations are scarce. Junbang Liang and co-authors flip the approach: first learn to generate videos of robot behaviour, then read actions out of the imagined footage. Their Video Policy framework trains video and action generation together, end to end, inside one modular system.
Policies drawn from generated video coped with unseen objects, backgrounds, and tasks in both simulation and the real world, using little demonstration data. Success tracked the quality of the generated video closely, and action-free video alone helped generalisation to new tasks. Against plain imitation learning, the method won clearly. The authors argue that big video models, built for entertainment, double as strong engines for robot learning.
Catalog record
| CREATOR | Junbang Liang Pavel Tokmakov Ruoshi Liu Sruthi Sudhakar, Paarth Shah Rares Ambrus Carl Vondrick |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Generators are Robot Policies Junbang Liang Pavel Tokmakov Ruoshi Liu Sruthi Sudhakar, Paarth Shah Rares Ambrus Carl Vondrick Abstract Despite tremendous progress in dexterous manipulation, current visuomotor policies remain fundamentally limited by two challenges: they struggle to generalize under perceptual or behavioral distribution shifts, and their performance is constrained by the size of human demonstration data. In this paper, we use video generation as a proxy for robot policy learning to address both limitations simultaneously. We propose Video Policy, a modular framework that combines video and action generation that can be trained end-to-end. Our results demonstrate that learning to generate videos of robot behavior allows for the extraction of policies with minimal demonstration data, significantly improving robustness and sample efficiency. Our method shows strong generalization to unseen objects, backgrounds, and tasks, both in simulation and the real world. We further highlight that task success is closely tied to the generated video, with action-free video data providing critical benefits for generalizing to novel tasks. By leveraging large-scale video generative models, we achieve superior performance compared to traditional behavior cloning, paving the way for more scalable and data-efficient robot policy learning. Figure 1: Video Generation as a Proxy for Robot Policy Learning. Given an initial observation and a natural language task prompt, our model generates a video of a robot executing the task (top) jointly with generating robot actions via a separate diffusion network (middle). This modular design enables learning from action-free video data and improves generalization to unseen scenarios, offering a scalable and sample-efficient alternative to traditional behavior cloning. Keywords: Behavior Cloning, Video Generation, Representation Learning 1 Introduction The fundamental challenge in visuomotor policy learning today is to create a robot system that robustly generalizes to new situations. Methods such as behavior cloning [ 1 , 2 ] have achieved impressive performance in many tasks ranging from pick-and-place operations to complex dexterous manipulation [ 3 , 4 , 5 ] . However, state-of-the-art methods are often unable to transfer learned behaviors to novel situations, failing to generalize across distribution shifts ranging from simple variations (e.g., color changes) to more complex challenges such as adapting to entirely new tasks [ 6 , 7 , 8 ] . While fields like computer vision and natural language processing have tackled these issues by collecting increasingly larger training sets to capture all variations [ 9 , 10 , 11 ] , robot action and human demonstration data are expensive to collect, making real-world generalization challenging. Generative video models [ 12 ] , such as OpenAI’s Sora [ 13 ] , are a promising direction for addressing these generalization challenges. Trained on massive datasets capturing diverse dynamics, situations, and physical interactions, they potentially encode powerful priors about how objects move and how actions affect the world. Thus, there is a strong interest in extracting robot policies from these foundation models to create robust control systems that can generalize to diverse scenarios and tasks. However, prior work along this direction has faced a fundamental trade-off: methods either rely on hand-crafted decoding mechanisms (such as tracking) [ 14 ] that limit expressivity and fail to capture the model’s full knowledge, or they use learned action decoders that introduce their own generalization gaps due to limited demonstration data [ 15 ] . Very recently, Hu et al. [ 16 ] have proposed a flexible approach for combining video generation and policy learning, but their work lacks a detailed analysis of the interplay between the two objectives. This paper demonstrates that video generative models are robot policies that generalize behavior to both visual and task distribution shifts. Our proposed model, Video Policy , is illustrated in Figure 1 . We systematically study video diffusion models and demonstrate that, as long as video generative models synthesize accurate videos of robot behavior, then learned decoders only need a surprisingly small amount of demonstration data to learn to map videos into actions that a robot can directly execute. Remarkably, we find that a small decoder can be trained to generalize to unseen tasks, suggesting that the video generative model is serving as the policy, while the decoder primarily serves as an interface rather than learning the task policy itself. Since the policy is strongly based on the video synthesized by a large-scale generative model, and the video model has been trained on a diverse set of videos, the approach generalizes to new objects, scenes, and new tasks with less training data than existing methods. In both simulation and real-world experiments, we show that using strong video priors improves performance compared to behavior cloning methods. Additionally, through detailed ablation studies, we provide insights into crucial design choices, guiding future research directions for capitalizing on generative video models for robot learning. We will release our code, models and implementation details for reproducibility. 2 Related Works Behavior Cloning. Behavior cloning (BC) is a widely adopted method in manipulation tasks, where policies are learned from demonstrations using supervised learning. Initial BC approaches focused on end-to-end models that mapped states directly to actions, yet these methods often encountered difficulties with multimodal behavior and tasks that require high precision [ 17 , 18 , 19 ] . To overcome these challenges, later research investigated Energy-Based Models (EBMs), which determine actions by minimizing an energy function during sequence optimization [ 20 , 21 , 22 , 23 ] . More recently, conditional generative models have emerged as a promising alternative, effectively capturing diverse demonstration behaviors and improving task success rates [ 2 , 24 , 25 ] . Unlike earlier models, our approach is based on video diffusion models with an extra action diffusion head to predict robot action jointly with pixels. Visual Pretraining for Policy Learning. There has been extensive research on pre-training perception models within visuomotor policies to achieve more robust visual representations. One commonly used objective is video prediction, where the model learns to forecast future frames from current observations, thereby capturing the dynamics and causal relationships essential for physical interactions [ 26 , 27 , 28 , 29 , 30 ] . Other popular self-supervised techniques include contrastive learning [ 27 , 31 , 32 , 10 ] and masked autoencoding [ 33 , 34 ] , which both contribute to developing strong visual features for robotics. Additionally, some studies have focused on learning a generalizable reward function through visual pretraining to support reinforcement learning tasks [ 35 , 36 , 37 ] . Video Models for Decision-Making. Early contributions in video generation [ 38 , 39 , 26 ] set the stage for employing generative models to predict future frames in a sequence. With recent advances in text-to-video generative models [ 13 , 12 , 40 , 41 , 42 ] , there is renewed interest in harnessing internet-scale video data for robotics. One line of research leverages video generative models as a form of world simulation, predicting future video sequences conditioned on actions [ 43 , 15 , 13 ] . Meanwhile, another approach utilizes video-language models to aid in long-horizon planning [ 44 , 45 , 46 ] . Several works also explore the joint pixel and action diffusion architecture [ 16 , 47 , 48 , 49 , 50 ] .