ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos

Xiaodong Wang, Peixi Peng · 2025

Read ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos by Xiaodong Wang, Peixi Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18650.txt.

Introduction

Xiaodong Wang and Peixi Peng published a research paper in 2025 on a driving world model that predicts future video and future driving actions together. Self-driving systems must watch the road, imagine what comes next, and choose moves. Earlier video models needed the actions handed to them, so they could not plan.

Their model learns hidden action patterns from the recent past, then rolls both the scene and the actions forward in step. A diffusion module handles how the scene itself evolves. On a large public driving set, the model topped rivals in video consistency and action accuracy. It also produced long, plausible futures, which planners need for rehearsing rare events.

Catalog record

CREATORXiaodong Wang, Peixi Peng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos Xiaodong Wang, Peixi Peng Abstract Real-world driving requires people to observe the current environment, anticipate the future, and make appropriate driving decisions. This requirement is aligned well with the capabilities of world models, which understand the environment and predict the future. However, recent world models in autonomous driving are built explicitly, where they could predict the future by controllable driving video generation. We argue that driving world models should have two additional abilities: action control and action prediction. Following this line, previous methods are limited because they predict the video requires given actions of the same length as the video and ignore the dynamical action laws. To address these issues, we propose ProphetDWM, a novel end-to-end driving world model that jointly predicts future videos and actions. Our world model has an action module to learn latent action from the present to the future period by giving the action sequence and observations. And a diffusion-model-based transition module to learn the state distribution. The model is jointly trained by learning latent actions given finite states and predicting action and video. The joint learning connects the action dynamics and states and enables long-term future prediction. We evaluate our method in video generation and action prediction tasks on the Nuscenes dataset. Compared to the state-of-the-art methods, our method achieves the best video consistency and best action prediction accuracy, while also enabling high-quality long-term video and action generation. 1 Introduction World models are expected to understand the environment and predict the future conditioned on given actions [ 21 , 10 ] . Early works often model the world models in latent feature space and have been validated on games or simulators [ 7 , 6 ] . Recently, inspired by the success of video generation [ 25 ] , several cutting-edge approaches [ 17 , 35 , 8 , 38 ] aim to build driving world models in an explicit way. That is, they could predict the future by controllable driving video generation and have shown promising results. Method Video Prediction Action Control Action Prediction One-stage DriveGAN [ 19 ] ✓ DriveDreamer [ 33 ] ✓ ✓ ✓ Drive-WM [ 35 ] ✓ ADriver-I [ 18 ] ✓ ✓ GenAD [ 38 ] ✓ Vista [ 8 ] ✓ ✓ Ours ✓ ✓ ✓ ✓ Table 1 : Driving world model comparison. Figure 1 : The comparison of video-action generation paradigms. We compare our method with previous methods that can perform action prediction. ADriver-I [ 18 ] and DriveDreamer [ 33 ] belong to two-stage methods, because they need first predict the video, and then predict the subsequent actions, and their video prediction are conditioned on corresponding actions. In contrast, our method learns from the given action sequence and current observation and jointly predicts future actions and video in a one-stage manner. Thanks to the development of diffusion models [ 15 , 27 ] , video prediction for driving scenarios is not very hard [ 3 , 19 , 35 , 18 , 38 ] . Besides, we argue that the driving world model should have two additional abilities: 1) Action control: different from pure video generation, world models are more concerned with the counterfactual reasoning of actions. That is, the generated videos should exactly reflect the future differences of various given actions; 2) Action prediction: The world model is not only used to visualize the future as video generation methods do, but also plan the near future action simultaneously. The accurate action prediction is even more important in autonomous driving tasks. Following this line, a comparison of recent driving world models is summarized in Table 1 , where the action refers to low-level driving action (such as steering angle and speed). Specifically, models [ 8 , 35 , 19 ] only focus on video generation, i.e., a kind of conditional prediction, given commands, trajectories, or low-level actions of every frame in the future, and ignore the action prediction. DriveDreamer [ 33 ] and ADrive-I [ 18 ] are more similar to this work. However, they separate the predictions of video and action, and both predict video first and then use the predicted video to predict action. We argue this paradigm should be improved for two reasons: Firstly, the length of the first video generation must be the same as the length of the given actions. Once the latter is limited, the long prediction must generated by multiple turns, and error accumulation may occur. Secondly, these methods only utilize the action sequence as a condition and ignore the dynamical action laws, which exhibit why such an action sequence is produced. Hence, as shown in Figure 1 , we propose a novel method that enables the driving world model to jointly learn to predict future actions and videos from given actions and observations. To this end, we consider the controllable driving task to be a finite-horizon, partially observable Markov decision process (POMDP) [ 28 ] . Video in the driving process is considered to be observation, and low-level driving actions are considered to be actions performed by agents. To model the POMDP, we construct a driving world model named ProphetDWM that predicts future states and actions based on a sequence of given states and actions. Similar to most world models in latent feature space [ 23 , 13 , 12 ] , we also design two important components to learn the states representation and latent actions. Firstly, we propose an action module that aims to learn the latent action from the present moment to the future period. The action module is lightweight and only stacked with MLP (Multilayer Perceptron). It could not only learn the dynamics information from given actions but also incorporate the states of current observation to learn a sequence of latent actions, and the latent actions are decoded into low-level actions, which are supervised by the ground-truth actions. Second, we leverage the state-of-the-art diffusion models as a transition model to learn the state distribution, where the states are converted from image observations by a VAE (Variational Autoencoder) encoder. Besides the self-attention among the sequence states in diffusion models, we design a short pathway that injects the state context to the hidden states inside the diffusion model, i.e., the latent action extracted from the action module, which aims to promote the quality and consistency of predictions. Finally, the two components are trained jointly to predict the actions and videos. To sum up, the major contributions of this paper are: • We propose ProphetDWM, an end-to-end driving world model that jointly learns to predict future actions and video. It is not a conditional low-level action-to-video model, but is trained by learning latent actions given finite states and predicting action and video. • The world model has an action module to learn latent actions from the present moment to the future period by giving the action sequence and observations and a diffusion-model-based transition model to learn the state distribution. The world model is trained to predict the actions and videos jointly. • We evaluate ProphetDWM in video generation and action prediction tasks on Nuscenes. Compared with state-of-the-art methods, our method achieves the best video

Read the complete text