DeepVerse: 4D Autoregressive Video Generation as a World Model
Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, Tong He · 2025
Read DeepVerse: 4D Autoregressive Video Generation as a World Model by Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, Tong He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01103.txt.
Introduction
Junyi Chen and a large team of co-authors wrote this 2025 paper on a video generator that remembers the shape of its world. World models let an agent predict what happens next and plan, but most predict only pictures, ignoring the hidden geometry underneath. Errors then pile up fast and scenes drift out of sync.
Their model, DeepVerse, predicts in four dimensions, folding the geometry estimated at each step into the next prediction, guided by the agent's actions. The explicit shape constraints capture richer links between space and time, cutting drift over long runs. A geometry-aware memory keeps distant parts of a scene consistent when revisited. Across varied scenarios the system produced longer, more accurate, and more believable futures than picture-only rivals.
Catalog record
| CREATOR | Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, Tong He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepVerse: 4D Autoregressive Video Generation as a World Model Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, Tong He Abstract World models serve as essential building blocks toward Artificial General Intelligence (AGI), enabling intelligent agents to predict future states and plan actions by simulating complex physical interactions. However, existing interactive models primarily predict visual observations, thereby neglecting crucial hidden states like geometric structures and spatial coherence. This leads to rapid error accumulation and temporal inconsistency. To address these limitations, we introduce DeepVerse, a novel 4D interactive world model explicitly incorporating geometric predictions from previous timesteps into current predictions conditioned on actions. Experiments demonstrate that by incorporating explicit geometric constraints, DeepVerse captures richer spatio-temporal relationships and underlying physical dynamics. This capability significantly reduces drift and enhances temporal consistency, enabling the model to reliably generate extended future sequences and achieve substantial improvements in prediction accuracy, visual realism, and scene rationality. Furthermore, our method provides an effective solution for geometry-aware memory retrieval, effectively preserving long-term spatial consistency. We validate the effectiveness of DeepVerse across diverse scenarios, establishing its capacity for high-fidelity, long-horizon predictions grounded in geometry-aware dynamics. Figure 1: We introduce DeepVerse, an interactive world model grounded in 4D autoregressive video generation. By establishing a 4D spatiotemporal distribution of the world, DeepVerse enables continuous and coherent 4D future prediction from merely a single input image, effectively modeling both spatial layouts and temporal dynamics simultaneously. 1 Introduction Interactive understanding of the physical world is a fundamental task for intelligent systems. World models, which aim to learn state transition functions from raw observations of external environments, provide essential predictive capabilities for intelligent agents, enabling them to imagine future states, evaluate possible actions, and navigate complex, dynamic scenarios. Recent progress in world models has demonstrated considerable potential in tasks such as visual simulation nvidia2025cosmosworldfoundationmodel , embodied navigation team2025aether , and manipulation zhen2025tesseract . Despite notable advancements in constructing effective world models, current online approaches feng2024thematrix ; valevski2024gamengen ; oasis2024 still suffer significantly from cumulative prediction errors and the forgetting issue. Addressing the above challenges is non-trivial. Most existing methods song2025historyguide ; xiao2025worldmem attempt to mitigate these issues by developing sophisticated techniques to efficiently incorporate historical frames. For instance, the recent FramePack zhang2025framepack compresses past frames into a fixed-length representation, thereby maintaining context within a transformer’s limited memory. However, these visual-centric strategies fundamentally overlook a critical aspect: videos inherently represent 2D projections of a dynamic 3D/4D physical world. Without explicit modeling of underlying geometric structures, models inevitably struggle to maintain long-term accuracy and consistency in visual predictions. To this end, we propose DeepVerse, the first autoregressive 4D world model trained on large-scale synthetic data with precise spatial labels. It explicitly incorporates geometric reasoning into online predictive modeling. By grounding visual forecasting in robust geometric representations, our method significantly enhances prediction accuracy, reduces drifting, and addresses the forgetting issue over extended temporal horizons. Specifically, our approach leverages a powerful autoregressive prior trained on large-scale real-world video data, capturing rich dynamic patterns and visual semantics. In parallel, we utilize extensive synthetic datasets that provide accurate ground-truth geometry supervision, including depth maps and camera poses. At each timestep, the model predicts the future state based not only on the previously generated RGB frames but also on the preceding geometric estimations. In doing so, the model greatly mitigates inherent issue in purely visual autoregressive systems, such as the scale ambiguity (figure 3 a), addressing the core issues of drifting and forgetting in conventional methods. Besides, to further address the forgetting issue, we introduce a geometry-aware memory read-and-write mechanism. We design a geometric memory module that compares the current geometry with historical observations and selects those with higher spatial overlap or structural similarity. These retrieved observations are then used as conditioning inputs for the current prediction step. This targeted memory retrieval allows the model to retain access to long-term contextual information without overwhelming the predictive pipeline, effectively reducing forgetting while maintaining spatial and temporal coherence over extended sequences. To sum up, our contributions can be summarized as follows: • We present DeepVerse, the pioneering introduction of an autoregressive 4D world modeling paradigm, establishing theoretical and practical guidelines for architectural configuration selection in future interactive world model development. • We innovatively incorporate 4D information into an autoregressive world modeling framework. By explicitly constructing the 4D world during the generation process, the proposed methodology demonstrates significant enhancement in visual consistency while effectively addressing scale ambiguity issues inherent to unimodal visual paradigms. • Building upon the DeepVerse framework’s capability for concurrent spatial distribution modeling, we have engineered a spatial memory mechanism to enhance long-term temporal consistency in generated sequences, thus establishing a robust framework for maintaining spatiotemporal continuity in autoregressive generation processes. 2 Method To provide a comprehensive illustration of our method, we elaborate on our problem formulation in section 2.1 and discribe the methodology for tailoring model architecture in section 2.2 . The construction of training datasets is systematically addressed in Section 2.3 . Finally, Section 2.4 demonstrates DeepVerse’s operational workflow during the inference phase. Figure 2: Our framework. The inputs to DeepVerse consist of: (1) a sequence of m consecutive 4D observations encompassing current and recent estimated states; (2) spatial conditions retrieved from a global memory pool through the selective mechanism \psi ; (3) textually specified control signals. The system subsequently generates k temporally coherent 4D future observations, which are automatically archived into the global memory repository for persistent world state tracking. 2.1 Problem Formulation World models aim to learn the transition function P(s_{t+1}|a_{t},s_{t}) in a Markov Decision Process (MDP), where s_{t} denotes the environment state at timestamp t . However, many real-world applications are Partially Observed MDPs (POMDPs) where s_{t} is latent. While prior work bruce2024genie ; valevski2024gamengen ; oasis2024 often used visual observations v_{t} directly, these provide an incomplete, non-Markovian signal. To address this, DeepVerse introduces a composite 4D state representation as a more informative proxy for s_{t} : \hat{s}_{t}=(v_{t},g_{t}). (1) Here, v_{t} is the visual observation, and g_{t} encapsulates geometric information, specifically relating to camera viewpoint c_{t} and depth d_{t} . This \hat{s}_{t} allows for a local 3D geometric representation, aiming to better approximate the hidden state s_{t} than v_{t} alone. A sequence of these \hat{s}_{t} representations forms a richer 4D data stream (3D space + time). While \hat{s}_{t} is fundamentally an enriched, structured observation , it is referred to as DeepVerse’s ‘state representation’ for operational simplicity. To mitigate drift from non-Markovian observations in POMDPs, DeepVerse employs an adaptive memory architecture, framing the task as a sequential auto-regressive prediction: f_{\theta}=P\left(\hat{s}_{t+1:t+k}\mid a_{t},\hat{s}_{t},\hat{s}_{t-m:t-1},\psi\left(\hat{s}_{0:t-m-1}\right)\right). (2) The model f_{\theta} in DeepVerse takes as inputs the action a_{t} , the current composite state representation \hat{s}_{t} , and the m most recent past representations \hat{s}_{t-m:t-1} . A selective mechanism incorporates an old representation \hat{s}_{i}=\psi(\hat{s}_{0:t-m-1}) only if it is statistically significantly correlated with \hat{s}_{t} . 2.2 Model Components 4D Representation. As formulated in Eq. 1 , DeepVerse employs a 4D representation for state estimation. Specifically, each g is a tensor with dimensions matching the input image, where each pixel stores a 3D coordinate wang2024dust3r . Following Aether team2025aether , we decompose 3D coordinates into depth d_{t} and viewpoint components. This allows depth information to be