Unified Video Action Model

Shuang Li Yihuai Gao Dorsa Sadigh Shuran Song · 2025

Read Unified Video Action Model by Shuang Li Yihuai Gao Dorsa Sadigh Shuran Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00200.txt.

Introduction

Li, Gao, Sadigh, and Song introduced a unified model in twenty twenty-five that learns from robot video and robot actions at the same time. Video shows a robot what a scene looks like, while action records show how the scene changes, so each stream can teach the other.

Their model, called UVA, learns one shared hidden code for video and action together. Two small decoder heads then split the code back into pictures or movements. Because the action head can skip making video, it answers fast while staying accurate. Training with parts of the input hidden lets one model handle many jobs: choosing actions, predicting what an action will do, and imagining future video. Tests across many robot tasks showed it matched tools built for a single job. The design points toward one general model for robot behavior.

Catalog record

CREATORShuang Li Yihuai Gao Dorsa Sadigh Shuran Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Video Action Model Shuang Li Yihuai Gao Dorsa Sadigh Shuran Song Abstract A unified video and action model holds significant promise for robotics, where videos provide rich scene information for action prediction, and actions provide dynamics information for video prediction. However, effectively combining video generation and action prediction remains challenging, and current video generation-based methods struggle to match the performance of direct policy learning in action accuracy and inference speed. To bridge this gap, we introduce the Unified Video Action model (UVA), which jointly optimizes video and action predictions to achieve both high accuracy and efficient action inference. The key lies in learning a joint video-action latent representation and decoupling video-action decoding. The joint latent representation bridges the visual and action domains, effectively modeling the relationship between video and action sequences. Meanwhile, the decoupled decoding, powered by two lightweight diffusion heads, enables high-speed action inference by bypassing video generation during inference. Such a unified framework further enables versatile functionality through masked input training. By selectively masking actions or videos, a single model can tackle diverse functions beyond policy learning, such as forward and inverse dynamics modeling and video generation. Via an extensive set of experiments, we demonstrate that UVA can serve as a general-purpose solution for a wide range of robotics tasks without compromising performance compared to methods tailored for specific applications. Results are best viewed on our website . Fig. 1 : Unified Video Action Model. (a) UVA features a joint video-action latent representation and decoupled video-action decoding. The joint latent representation effectively models the underlying relationships between video and action sequences, while the decoupled diffusion enables high-speed action inference by bypassing video generation. (b) By leveraging masked training, UVA supports flexible input-output combinations for actions and videos. This versatility allows the model to function as a robot policy, video model, forward or inverse dynamics model, or even a combined policy and video planner—all within a unified framework. I Introduction A unified video and action model that jointly learns an agent’s actions and their effects on visual observations holds great promise for robotics – videos provide rich environmental context for predicting actions, while actions reveal how interactions drive visual changes, enabling more accurate modeling of real-world dynamics. However, despite its promise, previous approaches have often failed to fully realize this potential. A key challenge lies in the inherent mismatch between the requirements of action and video generation. Action modeling demands high temporal speed to capture dense, fine-grained motions, while video generation requires high spatial resolution to produce high-fidelity visual outputs, which often results in slower processing speeds. Previous policy learning approaches have struggled to balance these conflicting requirements, often focusing on one aspect at the expense of the other. For instance, action only methods like [ 10 , 51 , 25 ] bypass video generation entirely. While such approaches reduce computational complexity, they overlook the benefits of video generation – adding observation supervision helps the model learn scene dynamics, which reduces overfitting to action history and enhances robustness to visual disturbances. On the other hand, video generation methods such as [ 14 , 28 ] often first generate high-resolution videos and then predict actions based on the generated videos. While this hierarchical approach can utilize existing video models, it also introduces significant drawbacks, including slower processing speeds and the propagation of errors from the generated video into action prediction. To address these limitations, we propose UVA , a Unified Video and Action Model designed to simultaneously model videos and actions – capturing the underlying interactions between visuals and actions to enhance task understanding, while maintaining high-speed action prediction during inference. We propose the following three design choices to achieve this: 1) Unified Latent Video-Action Representation: UVA introduces a unified latent representation that integrates both visual and action data. Unlike traditional video generation based policy methods which rely on a hierarchical video and action generation, UVA is trained simultaneously with supervision from both video and action data. This enables the model to capture the intricate dynamics shared between the visual and action domains with reduced computational overhead. Utilizing the rich scene information encoded in the latent representation unlocks UVA’s superior performance in understanding complex environments and delivering precise action predictions. 2) Decoupled Video-Action Diffusion for Fast Inference: To further enhance efficiency and achieve inference speed comparable to action-only methods, UVA decouples video generation from action prediction. During training, the model employs two lightweight diffusion heads to decode video observations and actions from the unified latent space. At inference, this decoupling allows the system to bypass video generation entirely, directly utilizing the latent representation for fast action prediction. This design enables real-time policy deployment without sacrificing performance, as it still retains the rich representations learned during training from both visual motions and robot action trajectories. 3) Mask Training for Flexibility: The ability to predict both videos and actions through unified representations further unlocks the potential to perform a diverse set of functions using masked training. UVA can handle versatile functions that go beyond traditional policy learning by masking inputs and outputs as needed, as illustrated in Figure 1 . This versatility enables the model to tackle complex scenarios, such as operating as a forward or inverse dynamics model, learning effectively from video-only datasets where action labels are unavailable, or simultaneously performing both low-level control and high-level planning. We evaluate UVA on seven publicly available benchmarks to assess its diverse capabilities. UVA outperforms or matches state-of-the-art approaches, demonstrating particularly strong performance in multi-task settings. For instance, UVA outperforms the best baseline by 20% in success rate on PushT Multitask [ 10 , 16 ] and by 5% on Libero10 [ 30 ] . The experiments show that UVA can serve as a general-purpose framework for different robotics tasks without compromising performance compared to methods tailored for specific applications. In sum, the unified video action model is: • Capable: UVA matches the state-of-the-art approaches that are tailored for robot policy learning [ 10 , 25 ] or planning [ 14 ] , especially for multi-task learning. • Practical: The use of decoupled diffusion heads eliminates the need for video generation during policy inference, and the use of lightweight diffusion heads reduces the costs of the denoising process. As a result, UVA achieves a similar speed as Diffusion Policy [ 10 ] , making it practical for robot applications. • Versatile: Beyond policy learning, UVA can also serve as a forward dynamics model for planning, as an inverse dynamics model to generate actions, a video generation model, or a combined policy and video planner. II Related Work There is a rich literature on video generation where the dominant approach includes diffusion-based methods [ 37 , 21 , 39 , 3 , 23 , 22 , 4 , 18 , 40 , 38 ] or autoregressive-based methods [ 46 , 50 , 45 , 13 , 17 , 47 ] . Our model utilizes the ideas from both of these techniques, as we will discuss in section III . Another line of related work involves the use of masked training in robot learning. In this section, we will review both video generation and masked training approaches used in robotics. Video Generation for Policy Learning:

Read the complete text