Learning 3D Persistent Embodied World Models
Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Dit-Yan Yeung, Chuang Gan · 2025
Read Learning 3D Persistent Embodied World Models by Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Dit-Yan Yeung, Chuang Gan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05495.txt.
Introduction
Siyuan Zhou and six colleagues give embodied agents a world model that remembers rooms it has already imagined, in a paper published in two thousand twenty-five. An agent that can simulate the future can plan, but video-based world models forget anything outside the current frame, so their long-horizon dreams contradict themselves.
The new model keeps an explicit memory of previously generated content. At each step a video diffusion model predicts the agent's future observations, in color and depth. Those predictions are folded into a persistent three-dimensional map of the environment. Later generation is conditioned on that map, so revisited places look as they did before. Consistency over long horizons is the whole point. Agents could then plan routes through partially seen buildings without the world reshuffling behind them. The work targets robots that must act, not just watch.
Catalog record
| CREATOR | Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Dit-Yan Yeung, Chuang Gan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning 3D Persistent Embodied World Models Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Dit-Yan Yeung, Chuang Gan Abstract The ability to simulate the effects of future actions on the world is a crucial ability of intelligent embodied agents, enabling agents to anticipate the effects of their actions and make plans accordingly. While a large body of existing work has explored how to construct such world models using video models, they are often myopic in nature, without any memory of a scene not captured by currently observed images, preventing agents from making consistent long-horizon plans in complex environments where many parts of the scene are partially observed. We introduce a new persistent embodied world model with an explicit memory of previously generated content, enabling much more consistent long-horizon simulation. During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent. This generation is then aggregated into a persistent 3D map of the environment. By conditioning the video model on this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world. Finally, we illustrate the efficacy of such a world model in downstream embodied applications, enabling effective planning and policy learning. Figure 1 : 3D Persistent Video Generation. Given the context video that defines the top-left 3D scene, the baseline world model deviates from this layout and introduces contradictory elements. In contrast, with 3D memory, our approach preserves observed structures and generates content consistent with the original context. 1 Introduction By training on vast and diverse datasets from the internet, large video generation models have demonstrated impressive capabilities that expand the horizons of computer vision and AI [ 38 , 2 , 5 , 13 ] . Such models are especially useful in embodied settings, where they can serve as world models, enabling simulation of how the dynamics of the world will evolve given actions [ 36 , 6 ] . Such an embodied world model can then significantly benefit tasks like path planning and navigation, enabling agents to make decisions based on simulated interactions before acting in the real world and enabling agents to be purely trained on simulated real-world interactions. However, a fundamental challenge exists for embodied world models: the underlying state of the world is represented as a single image or chunk of images, preventing existing world models from consistently simulating full 3D environments where much of the generated world remain unobserved at any given moment [ 17 ] . As a result, models are myopic and often generate new elements that conflict with its historical context—thereby harming the internal consistency required for real-world fidelity. These contradictions undermine the world models’ capacity to track evolving environments and plan over extended horizons, ultimately constraining their utility for navigation, manipulation, and other tasks demanding stable, long-term interaction. As illustrated in Figure 1 , given a context video defining a 3D scene, the world model without 3D memory fails to retain the unobserved regions once they move out of view. Compared to the ground truth frames, not only do items like the painting and table disappear, but the room structure turns into an entirely different space—showcasing a contradiction with the provided context. To solve this issue, we present Persistent Embodied World Model, an approach to learning an accurate embodied world model by explicitly incorporating a 3D memory into world models. Real-world scenes naturally unfold in three dimensions, so our model constructs a volumetric memory representation by populating 3D grids with DINO features [ 21 ] representing previously generated video frames, thereby capturing the spatial relationships within the environment. To enable accurately capture and generate the 3D geometry in an environment, our model processes and generates RGB-D data to preserve critical geometric cues in the memory’s 3D structure. To enable 3D consistent simulation, our model converts an agent’s actions into a corresponding relative camera pose change in the map, enabling effective retrieval from the 3D memory and ensuring consistent viewpoints and content across frames. By uniting 3D-structured memory, depth-aware generation, and precise camera control, our model not only predicts future observations under action control but also faithfully reconstructs past scenes with a high degree of spatial coherence. In Figure 1 , the 3D memory module allows our method to preserve previously observed 3D structure, maintaining coherent scene content throughout generation. In contrast, a model without the 3D memory progressively deviates from the given context. Our empirical results show significant enhancement for both the visual quality and consistency of video generation, highlighting the effectiveness of our proposed model and its associated memory mechanisms. In particular, these results confirm that incorporating a 3D memory promotes 3D persistence of embodied video generation, ensuring consistency with both previously generated and observed frames. Additionally, this persistence offers substantial advantages for the downstream robotic applications, including ranking the sampled action trajectories, planning with model predictive control, and policy training in the video simulators. In summary, our contributions are as follows: • We propose the Persistent Embodied World Models, an embodied world model system which incorporates 3D memory into video diffusion models, enabling persistent video generation. • We empirically demonstrate that our model offers significant improvement for both the visual quality and consistency of video generation. • We show how our model can be used in downstream embodied application such as planning and policy training. Figure 2 : Overview of our framework. (a) Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video. (d) The memory is incrementally updated after each video generation. (b, c) We train the memory blocks only and freeze DiT blocks in the second training stage. 2 Related work Embodied World Models Significant research efforts have developed generation models as world models in gaming [ 6 , 1 ] , autonomous driving [ 19 , 13 ] , and robotics [ 11 , 35 , 10 , 43 , 4 , 8 , 22 ] . Prior works [ 18 , 42 ] have primarily focused on training models using low-dimensional state and action representations on simulation data. These approaches leverage trajectory generation to enable robot planning. However, such methods face significant challenges in scaling to high-dimensional observation spaces. UniPi [ 11 ] , addresses this limitation by employing a video diffusion model to directly predict future visual frames subsequently coupled with an inverse dynamics model to infer action. Uni-Sim [ 35 ] and Genie [ 6 ] demonstrate the utility of world models for training agents by enabling policy learning without direct environmental interaction during training. However, existing world model frameworks in robotics remain largely restricted to simplified settings, such as table-top manipulation tasks, limiting their applicability in real-world scenarios requiring complex spatial reasoning. Most similar to our work, Navigation World Models [ 2 ] introduces a controllable video generation framework conditioned on navigation commands to simulate environment dynamics. While promising, this method lacks an explicit memory architecture, leading to scene inconsistency. Our framework integrates a memory mechanism that enables consistent long-horizon imagination, ensuring coherence with previously generated content. Video Generation and Memory A fundamental challenge in persistent video generation stems from computational memory limitations, which restrict the