RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation
Liudi Yang, Yang Bai, George Eskandar, Fengyi Shen, Mohammad Altillawi, Dong Chen, Soumajit Majumder, Ziyuan Liu, Gitta Kutyniok, Abhinav Valada · 2025
Read RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation by Liudi Yang, Yang Bai, George Eskandar, Fengyi Shen, Mohammad Altillawi, Dong Chen, Soumajit Majumder, Ziyuan Liu, Gitta Kutyniok, Abhinav Valada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22007.txt.
Introduction
Robots that learn from video need long, coherent demonstrations, but video generators drift and pile up errors when they extend a clip frame by frame. Liudi Yang, Yang Bai, and eight co-authors propose RoboEnvision, which plans the whole sequence first. The system breaks a broad goal into small step tasks and generates keyframes matched to each step. A second model then fills in the motion between each pair of keyframes, and a special attention module keeps objects and the scene consistent across frames.
A light policy model reads the finished video and recovers the robot's joint positions, so the generated footage can guide real actions. On two benchmarks the method topped earlier systems in video quality and consistency, and its policy beat prior models on long multi step tasks. The authors present it as a way to get useful long demonstrations without letting small errors compound.
Catalog record
| CREATOR | Liudi Yang, Yang Bai, George Eskandar, Fengyi Shen, Mohammad Altillawi, Dong Chen, Soumajit Majumder, Ziyuan Liu, Gitta Kutyniok, Abhinav Valada |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22007
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation Liudi Yang, Yang Bai, George Eskandar, Fengyi Shen, Mohammad Altillawi, Dong Chen, Soumajit Majumder, Ziyuan Liu, Gitta Kutyniok, Abhinav Valada Abstract We address the problem of generating long-horizon videos for robotic manipulation tasks. Text-to-video diffusion models have made significant progress in photorealism, language understanding, and motion generation but struggle with long-horizon robotic tasks. Recent works use video diffusion models for high-quality simulation data and predictive rollouts in robot planning. However, these works predict short sequences of the robot achieving one task and employ an autoregressive paradigm to extend to the long horizon, leading to error accumulations in the generated video and in the execution. To overcome these limitations, we propose a novel pipeline that bypasses the need for autoregressive generation. We achieve this through a threefold contribution: 1) we first decompose the high-level goals into smaller atomic tasks and generate keyframes aligned with these instructions. A second diffusion model then interpolates between each of the two generated frames, achieving the long-horizon video. 2) We propose a semantics preserving attention module to maintain consistency between the keyframes. 3) We design a lightweight policy model to regress the robot joint states from generated videos. Our approach achieves state-of-the-art results on two benchmarks in video quality and consistency while outperforming previous policy models on long-horizon tasks. I Introduction Video generation is a promising direction for solving complex robotic manipulation problems. The generated video conditioned on multi-step instructions can be used in planning and verification [ 1 , 2 ] , generalist policy learning [ 3 , 4 ] , and operation simulation [ 5 ] . The motivation for using video diffusion models to learn robotic tasks comes from their exceptional capabilities in photorealism, physics knowledge, and language understanding [ 6 , 7 ] . Recent adaptations of these models in embodied AI [ 5 , 3 , 8 ] have focused on generating short-term actions and relying on an autoregressive paradigm to extend them to the long horizon [ 6 , 9 , 10 , 11 , 1 ] , where future frames are predicted based on the last generated frames. However, chaining actions without regard to the long-horizon context can lead to accumulated errors in the execution and limited generalizability. The normal spatial-temporal attention widely used in video diffusion models degrades the frame consistency as the long context is not seen during autoregressive prediction. This can lead to inconsistencies in the generated frames, e.g., a different number of objects. Moreover, the observation space can shift from one task to another (going to another room to fetch a tool), further degrading the generation quality. Rather, the ability of the robot to plan, imagine a series of tasks, and execute them, given one high-level long-horizon instruction like ”clean the table” or ”make a pasta dish” is more desirable. Fig. 1: Top : Previous Works [ 3 , 8 ] predict short-horizon videos and estimate robot actions from them. Long horizon tasks are executed by cascading this approach sequentially along the time axis. Bottom : Our RoboEnvision model, breaks down a high-level instruction into small atomic instructions with a VLM, generates a frame aligned with each one, and interpolates between them. A policy model estimates the robot joints based on the keyframes and a few interpolated frames in between. In this work, we propose a novel pipeline of video generation for long-horizon robotic tasks. We raise the following question: ”How to generate a consistent long-horizon video reflecting the execution of multiple tasks given a single high-level instruction?” . Extending video diffusion models to this task features many challenges: (1) current models are limited in the number of generated frames, and autoregressive prediction introduces inconsistencies, (2) precise spatial understanding with language is hard to achieve, like ”push the red block to the right upper part of the blue block”, (3) generating small-sized objects and managing interactions between them while handling occlusions is challenging, (4) text/image-to-video models only follow the description in caption and cannot break it down into several tasks. These interactions are less prevalent in internet videos, making their accurate modeling more complex. While the video generation part is the main focus of this work, we also seek to answer the question: ”How to learn an action given the generated long-horizon video ?” Generating aligned videos and actions has several useful downstream applications. For instance, it can provide useful data augmentation to train policy models [ 12 , 13 , 14 , 4 , 1 ] for long horizon tasks, reducing the requirements for data collection. The generated videos can provide randomization across different levels, such as the image style, the environment, objects to be manipulated, and the order of executed tasks. Another downstream application is to directly use the generated videos and actions as a policy model. For this, we design a lightweight policy model that estimates robot actions from the generated videos and shows its effectiveness in executing complicated tasks. Contributions. Our key insight is to break down the long-horizon policy model into three stages, taking inspiration from how humans think: (1) breaking down the task into smaller sub-tasks using a VLM, (2) imagining the execution with a video generation model and (3) translating the generated videos to actions by predicting the robot’s joint configurations from the generated frames. As video diffusion models do not naturally extend to long horizon, we propose a novel two-stage approach: (a) a coarse keyframe diffusion model which predicts a sequence of keyframes, each representing the start or end of a sub-task (e.g. pick or place the object), (b) a fine filling diffusion model that interpolates between each two keyframes (see Fig. 1 ). While a similar coarse-to-fine approach appears in related work [ 3 ] , they are primarily designed for a short-horizon task, and the generated keyframes do not correspond to different sub-tasks. In contrast, our approach explicitly aligns each generated keyframe to a specific instruction from the VLM through an introduced keyframe-instruction cross-attention mechanism. However, we observe that inconsistencies remain in the generated keyframes, particularly regarding the disappearance or shape deformation of objects. This problem is primarily caused by the large motion of objects between consecutive keyframes, which are sampled from a long video at large intervals. To this end, we propose a semantics preserving attention module where we inject the first frame’s VAE features in the keyframe diffusion model. This ensures the preservation of object shape and location while allowing for a larger motion. Moreover, to validate the effectiveness of the generated long-horizon video, we design a lightweight transformer-based policy model to predict the robot joints’ angles and gripper state from the generated keyframes and some of the interpolated frames. To summarize, our main contributions are as follows: 1. We propose a novel pipeline for long-horizon video generation that integrates a VLM, coarse and fine video diffusion models, and a policy model. Our key insight is to generate keyframes aligned with simple instructions, each reflecting the end state of a task. 2. We design a novel attention module to enhance the geometric consistency of the objects and robot across the generated keyframes. 3. We achieve superior video quality and consistency compared to previous