Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving

Yunshen Wang, Yicheng Liu, Tianyuan Yuan, Yucheng Mao, Yingshi Liang, Xiuyu Yang, Honggang Zhang, Hang Zhao · 2025

Read Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving by Yunshen Wang, Yicheng Liu, Tianyuan Yuan, Yucheng Mao, Yingshi Liang, Xiuyu Yang, Honggang Zhang, Hang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23115.txt.

Introduction

Yunshen Wang and co-authors recast a self-driving vision task as a creative task for diffusion models, in a 2025 paper. A car must guess which parts of the space around it are filled, using camera input alone. Standard methods learn to label each cell directly, and they struggle with noise, blocked views, and complex scenes. Diffusion models instead learn the shape of real scenes and then generate a likely layout.

In wide tests, the generative approach beat the best direct methods, with the clearest gains in hidden or dim regions. The layouts looked more realistic and stayed more consistent across a scene. Better layouts also improved the planning steps that follow, such as choosing a safe path. The authors argue this makes diffusion models a practical base for real driving systems.

Catalog record

CREATORYunshen Wang, Yicheng Liu, Tianyuan Yuan, Yucheng Mao, Yingshi Liang, Xiuyu Yang, Honggang Zhang, Hang Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23115

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving Yunshen Wang, Yicheng Liu, Tianyuan Yuan, Yucheng Mao, Yingshi Liang, Xiuyu Yang, Honggang Zhang, Hang Zhao Abstract Accurately predicting 3D occupancy grids from visual inputs is critical for autonomous driving, but current discriminative methods struggle with noisy data, incomplete observations, and the complex structures inherent in 3D scenes. In this work, we reframe 3D occupancy prediction as a generative modeling task using diffusion models, which learn the underlying data distribution and incorporate 3D scene priors. This approach enhances prediction consistency, noise robustness, and better handles the intricacies of 3D spatial structures. Our extensive experiments show that diffusion-based generative models outperform state-of-the-art discriminative approaches, delivering more realistic and accurate occupancy predictions, especially in occluded or low-visibility regions. Moreover, the improved predictions significantly benefit downstream planning tasks, highlighting the practical advantages of our method for real-world autonomous driving applications. Fig. 1 : The diagram illustrates the occupancy data production process (a), the discriminative pipeline (b), and the generative pipeline (c). Black and orange circles highlight the deficiencies in the results from both the data production process and the discriminative pipeline, in contrast to the more reasonable results produced by the generative pipeline. I INTRODUCTION Vision-based 3D occupancy prediction is a task focused on estimating the semantic labels and occupancy states of each voxel in a scene from visual inputs, helping autonomous vehicles perceive their 3D environment with centimeter-level precision. Despite the recent progress in dataset, models and benchmarks [ 1 , 2 , 3 ] , accurately predicting the 3D occupancy grids is still a highly challenging task. Recent approaches in 3D occupancy prediction, which we refer to as discriminative methods, directly learn a mapping from images to occupancy grids and have become the de facto choice. However, there are inherent challenges in solving the 3D occupancy prediction task using discriminative methods: 1) The unique nature of predicting occupancy from visual inputs—such as the complex 3D structures, the intricate relationships between 3D labels and the existence of multimodal distributions—makes this task distinct and more challenging compared to other supervised learning problems. However, discriminative methods directly learn the mapping from images to occupancy, rather than estimating the underlying distributions, which prevents them from incorporating prior knowledge of 3D scenes. This often leads to unrealistic and inconsistent results (refer to the black and orange circle in Fig. 1 ). Moreover, these methods fail to capture the multimodal nature of the distributions, increasing the burden on downstream tasks. 2) Obtaining perfect occupancy map labels is almost infeasible. Existing benchmarks like KITTI360 [26] and Occ3D [39] rely on LiDAR scans from one or multiple traversals to create mesh reconstructions of scenes. However, partial observations and sensor noise lead to imperfect labels, which hinder effective model learning. Given these challenges, we believe that modeling occupancy prediction as a generative modeling problem offers a promising solution. By directly modeling the occupancy data distribution and performing conditional sampling, generative modeling captures the prior knowledge of complex 3D structures and semantics while naturally considering the inherent multi-modality of the occupancy prediction task. Furthermore, mainstream generative models, such as Diffusion Models exhibit inherent robustness to noise, which can mitigate the issue of harmful noise in occupancy labels. Moreover, generative models aim to model the underlying distribution of 3D world occupancy, rather than optimizing a direct mapping from images to occupancy. This results in better generalization across scenarios with varying sensor setups, such as input images that differ from the training set or lack certain information. Motivated by this, we explore how to leverage diffusion models for occupancy prediction, including investigating how to perform generative modeling and conditional sampling, and explore other interesting properties. Our extensive experiments demonstrate that generative modeling for occupancy prediction offers a series of powerful advantages over discriminative modeling. First, directly modeling the data distribution introduces a strong prior for 3D scene occupancy, enhancing the model’s perceptual capabilities, resulting in more realistic, consistent, and accurate outcomes. For regions with high uncertainty, such as those with insufficient observation, occlusion, or high levels of noise, our model exhibits superior perceptual capabilities. Such holistic, uncertainty-aware, and multimodal-considerate perception results also better support downstream planning tasks, as shown by our experimental results. Our key contributions are summarized as follows: • We frame occupancy prediction as a process involving generative modeling followed by conditional sampling, from which we summarized four appealing properties compare to discriminative counterpart. • We explore five key design aspects of utilizing conditional generative modeling for the occupancy prediction task. • Through extensive experiments, we demonstrate that incorporating diffusion models can significantly improve the performance of occupancy prediction. The occupancy features generated by our method also benefit downstream planning tasks. II Formulation We formulate the task of occupancy prediction using diffusion models [ 4 , 5 , 6 , 7 ] , which can express complex multimodal distributions and exhibit robustness to noise. Below, we provide an overview of diffusion models and their adaptation for occupancy prediction. II-A Diffusion Models Diffusion models progressively add noise to data and reverse this process to generate samples. For continuous data, the forward process adds Gaussian noise at each step: q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}), (1) where \beta_{t} controls the noise level at step t . The reverse process, parameterized by \theta , denoises the data: p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t),\sigma^{2}_{\theta}(t)\mathbf{I}), (2) For discrete data, diffusion models [ 6 , 7 ] replace Gaussian noise with a discrete corruption process, using a transition matrix \mathbf{Q}_{t} [ 7 ] . For discrete variables \mathbf{x}_{t},\mathbf{x}_{t-1}\in\{1,\ldots,K\} , the forward process becomes: q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\text{Cat}(\mathbf{x}_{t};\mathbf{p}=\mathbf{x}_{t-1}\mathbf{Q}_{t}), (3) The reverse process then predicts the discrete state transitions: p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\text{Cat}(\mathbf{x}_{t-1};\mathbf{p}=f_{\theta}(\mathbf{x}_{t},t)). (4) II-B Training Diffusion Models Starting from a sample \mathbf{x}_{0} , the marginal at step t is q(\mathbf{x}_{t}|\mathbf{x}_{0}) , with the posterior: q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})=\frac{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})q(\mathbf{x}_{t-1}|\mathbf{x}_{0})}{q(\mathbf{x}_{t}|\mathbf{x}_{0})}. (5) The reverse process is optimized by minimizing the KL divergence between the forward process and the predicted reverse process: \text{loss}=D_{KL}\left(q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})\,\|\,p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\right). (6) II-C Occupancy Prediction as Conditional Generation We adapt diffusion models to predict occupancy by modifying the output \mathbf{x} to represent occupancy grids and conditioning the reverse process on input multi-view images \mathbf{C} . This modifies Eq. ( 2 ) to: p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{C})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t,\mathbf{C}),\sigma^{2}_{\theta}(t)\mathbf{I}), (7) and for discrete models Eq. ( 4 ) becomes: p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{C})=\text{Cat}(\mathbf{x}_{t-1};\mathbf{p}=f_{\theta}(\mathbf{x}_{t},t,\mathbf{C})). (8) Fig. 2 : An overview of using diffusion models for occupancy prediction. A base BEV model is employed to encode the input into high-dimensional features, which serve as conditions for the diffusion models. The diffusion model is then able to model different representations of the occupancy grid data. TABLE I : Comparison of different occupancy representations for modeling with diffusion models. “DiffOcc(***)” denoting the adaptation of diffusion models to the specified representations denoted as “***”. Model BEVFormer DiffOcc(latent) DiffOcc(triplane) DiffOcc(discrete) mIoU 23.67 30.65 30.23 31.78 TABLE II : Comparison of mIoU for different guidance techniques and scales, obtained using discrete diffusion models. Guidance Tech Scale 0.5 Scale 1 Scale 2 Scale 3.5 CFG 27.34 31.78 31.98 32.45 CG 26.50 29.45 29.98 29.97 III Key Design Decisions III-A Denoiser Network Architecture We use a U-Net variant [ 8 ] as the denoiser network, trained to predict the clean mask x_{0} rather than directly predicting x_{t-1} , following the x_{0} parameterization approach. A point cloud segmentation network [ 9 ] is adapted for denoising, with modifications

Read the complete text