Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets

Chuning Zhu 2 , Raymond Yu 2 , Siyuan Feng 3 , Benjamin Burchfiel 3 , Paarth Shah 3 , and Abhishek Gupta 2 · 2025

Read Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets by Chuning Zhu 2 , Raymond Yu 2 , Siyuan Feng 3 , Benjamin Burchfiel 3 , Paarth Shah 3 , and Abhishek Gupta 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02792.txt.

Description

Chuning Zhu leads a team of six on this paper from two thousand twenty-five. It trains robots in one frame that is at once a policy, a world model, and a video maker, by pairing video and action diffusion.

Copy-the-expert learning starves for costly demos. Video of real acts is vast, but lacks action marks. Unified World Models run two diffusion flows in one transformer, one for frames and one for acts, each with its own clock. Turn the clocks and the same net acts, foretells, works backward, or dreams video. Pretraining on big mixed robot sets yields tougher, broader policies than pure copying. Action-free video can teach too, through the split clocks. Trials in sim and on real robots back the gains.

Catalog record

CREATORChuning Zhu 2 , Raymond Yu 2 , Siyuan Feng 3 , Benjamin Burchfiel 3 , Paarth Shah 3 , and Abhishek Gupta 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02792

Browse the full library · About the Tower of Babel