DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

Teli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising · 2026

Read DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control by Teli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10448.txt.

Description

Teli Ma and co-authors introduce DiT4DiT, a robot control model that learns video dynamics and actions together. Such models lean on static image and text pretraining, leaving physics to scarce action data, while video generators encode rich motion structure. DiT4DiT takes mid-denoising features from its video transformer as conditions for an action transformer. A dual flow-matching objective trains both parts jointly. It set leading scores on LIBERO, learned from far less data, and generalized well on a real Unitree G1.

Catalog record

CREATORTeli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62397
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10448

Browse the full library · About the Tower of Babel