Tes ser Act : Learning 4D Embodied World Models
Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, Chuang Gan · 2025
Read Tes ser Act : Learning 4D Embodied World Models by Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, Chuang Gan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20995.txt.
Introduction
Haoyu Zhen and six co-authors present TesserAct, a model that learns how a three-dimensional scene changes over time in response to a robot's actions. Their paper came out in 2025. Such a world model lets an embodied agent, meaning a robot acting in a physical space, predict the results of its actions before taking them.
The team enriched existing robot manipulation videos with depth and surface direction information from ready-made models. They then fine-tuned a video generation model to predict color, depth, and surface data together for each frame. An algorithm converts those predictions into a coherent four-dimensional scene. The resulting model supports new viewpoint synthesis and policy learning, meaning learning how to act, that beat earlier video-based world models by a clear margin.
Catalog record
| CREATOR | Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, Chuang Gan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20995
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tes ser Act : Learning 4D Embodied World Models Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, Chuang Gan Abstract This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent’s actions, providing both spatial and temporal consistency. We propose to learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos. This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent. Specifically, we first extend existing robotic manipulation video datasets with depth and normal information leveraging off-the-shelf models. Next, we fine-tune a video generation model on this annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for each frame. We then present an algorithm to directly convert generated RGB, Depth, and Normal videos into a high-quality 4D scene of the world. Our method ensures temporal and spatial coherence in 4D scene predictions from embodied scenarios, enables novel view synthesis for embodied environments, and facilitates policy learning that significantly outperforms those derived from prior video-based world models. https://TesserActWorld.github.io Figure 1 : We propose TesserAct, the 4D Embodied World Model, which takes an input image and text instruction to generate RGB, depth, and normal videos, reconstructing a 4D scene and predicting actions. Our model not only achieves strong performance on in-domain data (right) but also generalizes effectively to unseen scenes, novel objects (top left), and cross-domain scenarios (bottom left). † † footnotetext: ∗ Equal contribution. 1 Introduction Learned world models [ 21 , 67 , 77 , 64 ] , which simulate environmental dynamics, play a crucial role in enabling embodied intelligent agents. Such models enable flexible policy synthesis [ 15 , 40 ] , data simulation and generation [ 67 , 80 ] , and long-horizon planning [ 28 , 14 , 74 ] . However, while the physical world is three-dimensional in nature, existing world models operate in the space of 2D pixels. This limitation leads to an incomplete representation of spatial relationships, impeding tasks that require precise depth and pose information. For instance, without accurate depth and 6-DoF pose estimations, robotic systems struggle to determine the exact position and orientation of objects. Furthermore, existing 2D models can produce unrealistic results, such as inconsistent object sizes and shapes across time steps, which limits their use in data-driven simulations and robust policy learning. In this paper, we explore how we can instead learn a 4D embodied world model, TesserAct, which simulates the dynamics of a 3D world. This 4D embodied world model allows us to generate realistic 3D interactions, such as grasping objects or opening drawers, with a level of detail that traditional 2D-based models cannot achieve. By modeling spatial and temporal dimensions, our model provides the depth and pose information essential for robotic manipulation. However, the task of learning a 4D embodied world model is challenging as the dynamics of the world are extremely computationally expensive to train and learn, requiring models to generate outputs in three-dimensional space and time. To efficiently represent and predict the dynamics of the world, we propose a substantially more lightweight representation of the 4D world, consisting of predicting a sequence of RGB, depth, and normal maps of the scene. This combined representation accurately captures the appearance, geometry, and surface of a scene while being substantially lower dimensional than explicitly predicting world dynamics. Furthermore, such a representation shares substantial similarities to existing video models, allowing us to directly use the generative capabilities of existing video models to effectively construct our 4D world model. Given this intermediate representation, we present an efficient algorithm to reconstruct accurate 4D scenes from generated RGB-DN videos. For each frame, we use a combination of both depth and normal prediction to integrate a smooth 3D surface of the scene. We then use optical flow between generated frames to distinguish between background and dynamic regions in the reconstructed 3D scene across frames and introduce two novel loss functions to enforce consistency across scenes over time. As shown in Figure 1 , our 4D Embodied World Model enables the construction of high-fidelity 4D-generated scenes, facilitating strong-performance action planning for downstream tasks. A key challenge for training TesserAct is a lack of access to existing large-scale datasets with high-quality 4D annotations, or the RGB image, depth, and normal information needed to train our approach. To overcome this, we collect a 4D embodied video dataset consisting of synthetic data with ground truth depth and normal information and real-world data with estimated depth and normal information with off-the-shelf estimators. Overall, our paper has the following contributions: • We collect a 4D embodied video dataset with compact and high-quality RGB, depth, and normal annotations and learn a 4D embodied world model. • We present an algorithm to convert the generated RGB-DN video into high-quality 4D scenes coherent across both time and space. • Extensive experiments demonstrate our 4D embodied world model can predict high-fidelity 4D scenes and achieve superior performance in downstream embodied tasks compared to traditional video-based world models. Dataset Domain Depth Source Normal Source Embodiment # of videos RLBench [ 26 ] Synthetic Simulator Depth2Normal [ 2 ] Franka Panda 80k RT1 Fractal Data [ 6 ] Real Rolling Depth [ 31 ] Marigold [ 32 ] Google Robot 80k Bridge [ 61 ] WidowX 25k SomethingSomethingV2 [ 19 ] Human Hand 100k Table 1 : Overview of the 4D embodied video datasets. 2 Related Work Embodied Foundation Models A flurry of recent work has focused on constructing foundation models for general purpose agents [ 68 , 18 ] . One line of work has focused on constructing multimodal language models that operate over images [ 39 , 29 , 51 , 13 , 63 , 73 , 20 ] as well as 3D inputs [ 24 , 25 ] and output text describing the actions of an agent. Other works have focused on the construction of vision-language-action (VLA) models that directly output action tokens [ 7 , 34 , 76 ] . Both of the previous approaches aim to construct foundation model policies (over text or continuous actions). In contrast, our work aims to instead construct a foundation 4D world model for embodied agents, which can then be used for downstream applications such as planning [ 14 , 74 ] or policy synthesis [ 15 , 40 ] . Learning World Models Learning dynamics model of the world given control inputs has been a long-standing challenge in system identification [ 42 ] , model-based reinforcement learning [ 56 ] , and optimal control [ 81 , 4 ] . A large body of work focused on learning world models in the low dimensional state space [ 17 , 1 , 38 ] , which while being efficient to learn, is difficult to generalize across many environments. Other works have explored how world models may be learned over pixel-space images [ 12 , 21 , 22 , 11 , 43 ] , but such models are trained on simple game environments. With advances in generative