DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Teli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising · 2026
Read DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control by Teli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10448.txt.
Description
Teli Ma and co-authors introduce DiT4DiT, a robot control model that learns video dynamics and actions together. Such models lean on static image and text pretraining, leaving physics to scarce action data, while video generators encode rich motion structure. DiT4DiT takes mid-denoising features from its video transformer as conditions for an action transformer. A dual flow-matching objective trains both parts jointly. It set leading scores on LIBERO, learned from far less data, and generalized well on a real Unitree G1.
Catalog record
| CREATOR | Teli Ma Jia Zheng Zifan Wang Chunli Jiang Andy Cui, [0.8ex] Junwei Liang Shuo Yang, [1.2ex] Mondo Robotics HKUST(GZ) HKUST, [0.18cm] Corresponding author, Co-advising |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62397 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10448