Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models
Jianbiao Mei, Yu Yang, Xuemeng Yang, Licheng Wen, Jiajun Lv, Botian Shi, Yong Liu · 2025
Read Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models by Jianbiao Mei, Yu Yang, Xuemeng Yang, Licheng Wen, Jiajun Lv, Botian Shi, Yong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16729.txt.
Introduction
Jianbiao Mei and six co-authors present IR-WM, a world model for vision-based driving that predicts only how a scene changes. Most driving world models rebuild the whole future scene, wasting effort on static background. Their model instead keeps a strong view of the present and forecasts the residual, meaning the change since the last step.
The model builds a bird's-eye view from camera input, then uses the prior view as a base for action-aware change prediction. An alignment module corrects meaning and motion drift over time. The team also tests ways to link forecasting with planning. On the nuScenes benchmark, the implicit future states improved planning accuracy.
Catalog record
| CREATOR | Jianbiao Mei, Yu Yang, Xuemeng Yang, Licheng Wen, Jiajun Lv, Botian Shi, Yong Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models Jianbiao Mei, Yu Yang, Xuemeng Yang, Licheng Wen, Jiajun Lv, Botian Shi, Yong Liu Abstract End-to-end autonomous driving systems increasingly rely on vision-centric world models to understand and predict their environment. However, a common ineffectiveness in these models is the full reconstruction of future scenes, which expends significant capacity on redundantly modeling static backgrounds. To address this, we propose IR-WM, an Implicit Residual World Model that focuses on modeling the current state and evolution of the world. IR-WM first establishes a robust bird’s-eye-view representation of the current state from the visual observation. It then leverages the BEV features from the previous timestep as a strong temporal prior and predicts only the “residual”, i.e., the changes conditioned on the ego-vehicle’s actions and scene context. To alleviate error accumulation over time, we further apply an alignment module to calibrate semantic and dynamic misalignments. Moreover, we investigate different forecasting–planning coupling schemes and demonstrate that the implicit future state generated by world models substantially improves planning accuracy. On the nuScenes benchmark, IR-WM achieves top performance in both 4D occupancy forecasting and trajectory planning. I Introduction End-to-end autonomous driving models, which directly map raw sensor data to planning outputs, have become a dominant paradigm [ 1 , 2 , 3 , 4 ] . At its core, safe decision-making in autonomous driving hinges on the precise understanding of current and past status, along with the reliable anticipation of future events. Consequently, action-driven world models have emerged as a promising approach that bridges agent intentions and environmental dynamics through action-guided world generation and forecast-driven planning, thereby enabling more reliable decision-making. However, existing vision-centric world models for end-to-end planning often cast forecasting as full-scene reconstruction. Whether they are implicit models that produce low-dimension latents [ 5 , 6 ] or explicit models that predict high-dimension representations like videos and 4D occupancy [ 7 , 8 ] , they tend to regenerate the entire scene from scratch for both current and future states, wasting capacity on largely static backgrounds and constraining scene context encoding from past and current visual observation. This motivates a shift from regenerating the world to modeling the current state and its future changes, as shown in Fig. 1 , better allocating model capacity. Therefore, we introduce the Implicit Residual World Model, IR-WM , guided by auxiliary vision-centric 4D occupancy forecasting to enable action-guided world generation and forecast-driven planning. IR-WM first constructs an implicit bird’s-eye-view (BEV) representation of the current state from raw images. Instead of reconstructing full futures, it treats the previous BEV as a spatiotemporal prior and predicts only residual changes conditioned on ego trajectories. This residual design yields: (1) modeling efficiency by focusing capacity on scene context encoding from visual observation and salient context changes, e.g., dynamics and background changes, for forecasting; (2) improved dynamics understanding by emphasizing temporal differences; and (3) temporal coherence by grounding each prediction in the preceding state. We recover future states by adding residuals to the prior and applying an alignment module to correct semantic and dynamic misalignments, alleviating potential error accumulation during rollout. All BEV representations are supervised with semantic occupancy grids, enhancing semantic richness and geometric fidelity for reliable planning. Fig. 1: We model the current state and predict residual future changes, yielding a more effective allocation of model capacity. Moreover, we design different forecasting–planning coupling schemes to study how forecasting influences planning. Our insights are: (1) using occupancy forecasting to filter candidate trajectories is largely unnecessary, offering only marginal planning gains while adding latency; (2) implicit future state generated by world models substantially improves planning accuracy; and (3) action-conditioned world models can serve as effective observers, providing strong supervisory signals for trajectory generation. We evaluate IR-WM on nuScenes [ 9 ] for vision-centric 4D occupancy and trajectory planning. Under the OpenOccupancy benchmark [ 10 ] , IR-WM achieves consistent gains across 4D occupancy tasks and significantly reduces L2 error and collision rate relative to the baseline. Fig. 2: Overview of IR-WM. IR-WM comprises three parts: (a) Scene encoding from visual observations via a scene encoder; (b) future forecasting, performed in an implicit residual prediction manner with a streaming memory using an autoregressive future predictor; (c) Task decoding via occupancy head D_{o} and planning head D_{p} . “FA” denotes the feature alignment for alleviating error accumulation. Our main contributions can be summarized as follows: • We introduce IR-WM, a world model that models the current world state and its dynamics in a compact BEV space, providing a strong basis for vision-centric 4D occupancy forecasting and trajectory planning. • We investigate the impact of forecasting on planning, and exploit a semi-coupled design separating 4D occupancy and planning to enhance flexibility while maintaining planning accuracy. • Extensive experiments demonstrate the effectiveness of IR-WM. It achieves state-of-the-art performance in both 4D occupancy forecasting and trajectory planning. II Related Works II-A Driving World Models World models aim to simulate future environment states given past observations and actions. Image-based approaches [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] generate future driving videos using diffusion or transformer architectures, often conditioned on actions, BEV layouts, or HD maps to enhance realism. In contrast, volume-based methods [ 19 , 20 , 21 , 7 , 22 , 23 ] directly model 3D structures such as occupancy or point clouds, enabling spatially consistent future prediction. Unlike recent approaches, we construct a residual BEV world model with feature alignment, guided by 4D occupancy forecasting, which balances model capacity to effectively capture both the current state and its future changes. This design alleviates error accumulation, leverages vision-centric state modeling as a foundation for accurate future prediction, and strengthens downstream planning. II-B End-to-end Driving End-to-end models have emerged as a paradigm shift in autonomous driving, directly mapping sensor inputs to future trajectories [ 3 , 4 ] . Early approaches such as P3 [ 24 ] and ST-P3 [ 25 ] incorporated differentiable occupancy to ensure safe planning. More recent frameworks leverage BEV perception as an intermediate representation, including UniAD [ 3 ] , VAD [ 4 ] , and GraphAD [ 26 ] . Further advances focus on disentangled scene representations [ 27 ] , reducing annotation costs [ 28 , 5 ] , or improving efficiency through parallel or sparse architectures [ 29 , 30 , 31 ] . Our work explores the impact of implicit representation generated by world models on end-to-end planning. II-C Occupancy Prediction and Forecasting Occupancy prediction reconstructs the 3D state of the environment. LiDAR-based methods [ 32 , 33 , 34 , 35 ] complete sparse point clouds into dense voxel grids, while camera-based approaches [ 36 , 37 , 10 , 38 ] transform 2D features into 3D space, often enhanced with depth estimation [ 39 ] or multi-view aggregation [ 40 ] . Forecasting extends this task to the temporal domain, with LiDAR-driven approaches that predict future points or occupancy volumes [ 41 , 42 ] , and 4D occupancy forecasting of Cam4DOcc [ 43 ] and Drive-OccWorld [ 8 ] . This work leverages occupancy forecasting as a fine-grained regularization to learn compact world state representations and