Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, , Wenjun Zeng · 2025
Read Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method by Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, , Wenjun Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22973.txt.
Introduction
Bohan Li leads a team of twelve researchers working to generate realistic driving scenes for self-driving cars. Occupancy-based methods, which map space as filled or empty, give steady control across frames. Their weakness is a lack of labeled occupancy data, which is costly to make by hand.
The group built Nuplan-Occ, described as the largest semantic occupancy data set so far, from the NuPlan benchmark. They also present a unified method that creates high-quality scenes across sensors. The larger data set should help teams test perception and planning with more varied and consistent scenes.
Catalog record
| CREATOR | Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, , Wenjun Zeng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22973
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, , Wenjun Zeng Abstract Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. I Introduction The generation of high-quality driving scenes represents a promising avenue for advancing autonomous driving (AD), as it alleviates the significant resource demands associated with real-world data collection and annotation [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . Recent breakthroughs in generative models, particularly diffusion-based approaches [ 1 , 2 , 3 , 4 ] , have enabled the creation of highly realistic synthetic data [ 9 , 10 , 11 ] , thereby facilitating advancements in downstream tasks. Current driving scene generation works [ 11 , 12 , 13 , 14 ] commonly rely on layout conditions derived from coarse geometric labels, such as bird’s-eye-view (BEV) maps and 3D bounding boxes, to guide the scene generation process. The synthetic data produced through these methods is subsequently utilized to enhance the performance of downstream tasks, including BEV segmentation [ 15 , 16 , 17 ] and 3D object detection [ 18 , 19 , 20 , 21 , 22 ] . These driving scene generation models predominantly focus on producing data in a single format ( e.g. , RGB video) [ 11 , 23 , 12 , 24 ] , without fully leveraging the potential of generating data across multiple formats. This limitation restricts their applicability to a broad range of downstream tasks that rely on diverse sensor data, including RGB video and LiDAR point clouds in real-world scenarios [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 14 , 4 ] . Furthermore, existing methods typically attempt to model the real-world distribution using a single-step layout-to-data generation process based solely on coarse input conditions ( e.g. , BEV layouts or 3D bounding boxes) [ 12 , 24 , 32 ] . This direct learning approach often undermines the model’s ability to capture the intricate distributions inherent in real-world driving scenes ( e.g. , realistic geometry and appearance) and leads to suboptimal performance. \blacksquare other-ground \blacksquare vehicle \blacksquare bicycle \blacksquare pedestrian \blacksquare traffic-cone \blacksquare barrier \blacksquare construction-zones \blacksquare generic-object \blacksquare road \blacksquare road-line Fig. 1 : Overview of Nuplan-Occ dataset and the UniScenev2 pipeline. We introduce the largest semantic occupancy dataset to date, featuring dense 3D semantic annotations that contain \sim 19× more annotated scenes and \sim 18× more frames than Nuscenes-Occupancy [ 33 , 34 ] . Facilitated with Nuplan-Occ, UniScenev2 scales up both model architecture and training data to enable high-quality occupancy spatial expansion and temporal forecasting, as well as occupancy-based sparse point map condition for video generation, and sensor-specific LiDAR generation. To address these challenges, UniScene [ 6 ] proposes to utilize 3D semantic occupancy as an intermediate representation with rich semantic and geometric information to decompose complex autonomous driving scene generation tasks into hierarchical steps for high-quality multi-modal generation of semantic occupancy, video, and LiDAR data [ 35 , 36 , 25 , 30 , 31 , 14 , 4 ] . Within the framework, 3D semantic occupancy is first generated from BEV scene layouts and then utilized to guide the subsequent generation of video and LiDAR data. The generated semantic occupancy serves as an intermediate representation, guiding the subsequent generation of other output modalities with 3D structural details and semantic priors. However, the generation capabilities of UniScene remain constrained by limited scenario diversity and scale, akin to prior works [ 11 , 12 , 13 , 14 , 32 ] , which limits its practical utility for scalable downstream tasks. To address these limitations, we propose UniScenev2, a unified occupancy-centric framework for versatile 4D dynamic scene generation of semantic occupancy, video, and LiDAR data. Beyond UniScene [ 6 ] , which generates 3D semantic occupancy, multi-view video, and LiDAR data via a decomposed learning paradigm and hierarchical architecture, UniScenev2 overcomes its predecessor’s scalability constraints on the Nuscenes [ 37 ] dataset. By scaling both model architecture and training data, UniScenev2 achieves large-scale semantic occupancy generation and synthesizes corresponding multi-view videos and LiDAR point clouds, as shown in Figure 1 . Specifically, to enable efficient training across diverse autonomous driving scenarios, we construct Nuplan-Occ, a large-scale semantic occupancy dataset extending the Nuplan [ 38 ] benchmark with dense 3D semantic annotations. As detailed in Figure 1 and Table I , Nuplan-Occ contains \sim 19× more annotated scenes and \sim 18× more frames than Nuscenes-Occupancy [ 33 , 34 ] . Our automated annotation pipeline employs a Foreground-Background Separate Aggregation Strategy (FBSA) for dense reconstruction and precise semantic label assignment, which reconstructs dense semantic occupancy grids by separately aggregating foreground objects and background content from multi-frame LiDAR scans. This process involves point-based registration, denoising, neural kernel-based reconstruction [ 39 ] , and voxelization for precise semantic labeling. Dataset Type Surrounded View Modility #Sequence #Frames Volume Size Resolution(m) NNYUv2 [ 40 ] Indoor 1 C&D 0.5K 1.4K [240,240,14] - SceneNN [ 41 ] Indoor 1 C&D 100 - - - SynthCity [ 42 ] Indoor 1 C&D 9 - - - ScanNet [ 43 ] Indoor 1 C&D 1.5K 1.5K [62,62,31] - SemanticPOSS [ 44 ] Indoor 1 C&D - 3K - - SemanticKITTI [ 45 ] Outdoor 2 C&L 22 4K [256,256,32] [0.2,0.2,0.2] KITTI-360 [ 46 ] Outdoor Fisheye 2 C&L 11 90K [256,256,32] [0.2,0.2,0.2] SurroundOcc [ 34 ] Outdoor 6 C&L 1K 200K [200,200,16] [0.5,0.5,0.5] Occ3D [ 33 ] Outdoor 6 C&L 1K 200K [200,200,16] [0.4,0.4,0.4] OpenScene [ 47 ] Outdoor 8 C&L 1.8K 69K [200,200,16] [0.5,0.5,0.5] Nuplan-Occ (Ours) Outdoor 8 C&L 19K 3614K [400,400,32] [0.25,0.25,0.25] TABLE I : Comparison between Nuplan-Occ and other occupancy/LiDAR datasets. “Surrounded” represents surround-view image inputs. “View” means the number of image view inputs. “C”, “D”, and “L” denote camera, depth, and LiDAR, respectively. \blacksquare other-ground \blacksquare vehicle \blacksquare bicycle \blacksquare pedestrian \blacksquare traffic-cone \blacksquare barrier \blacksquare construction-zones \blacksquare generic-object \blacksquare road \blacksquare road-line Fig. 2 : The Nuplan-Occ provides dense semantic occupancy labels