HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving

Zehuan Wu 1∗ 🖂 {}^{\hrefmailto:[email protected]} Jingcheng Ni 1∗ Xiaodong Wang 3∗ Yuxin Guo 1∗ Rui Chen 1∗ Lewei Lu 1 Jifeng Dai 2,4 Yuwen Xiong 2 🖂 {}^{\hrefmailto:[email protected]} 1 Sensetime Research 2 Shanghai Artificial Intelligence Laboratory 3 Peking University 4 Tsinghua Unive · 2024

Read HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving by Zehuan Wu 1∗ 🖂 {}^{\hrefmailto:[email protected]} Jingcheng Ni 1∗ Xiaodong Wang 3∗ Yuxin Guo 1∗ Rui Chen 1∗ Lewei Lu 1 Jifeng Dai 2,4 Yuwen Xiong 2 🖂 {}^{\hrefmailto:[email protected]} 1 Sensetime Research 2 Shanghai Artificial Intelligence Laboratory 3 Peking University 4 Tsinghua Unive (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01407.txt.

Introduction

Zehuan Wu and co-authors present HoloDrive, a system that dreams up street scenes for self-driving cars in both camera views and laser scans at once. Real cars use both kinds of input, since each fills gaps in the other. Past makers built one kind at a time, so their scenes lacked that joined feel.

Bridge modules pass facts between the image maker and the scan maker in both directions. A depth-guessing branch in the image side keeps the lift into bird's-eye view from going astray. A staged training plan then stretches the system to predict future frames. On standard tests it beat top rivals by clear margins in scene quality.

Catalog record

CREATORZehuan Wu 1∗ 🖂 {}^{\hrefmailto:[email protected]} Jingcheng Ni 1∗ Xiaodong Wang 3∗ Yuxin Guo 1∗ Rui Chen 1∗ Lewei Lu 1 Jifeng Dai 2,4 Yuwen Xiong 2 🖂 {}^{\hrefmailto:[email protected]} 1 Sensetime Research 2 Shanghai Artificial Intelligence Laboratory 3 Peking University 4 Tsinghua Unive
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving Zehuan Wu 1∗ 🖂 {}^{\hrefmailto:[email protected]} Jingcheng Ni 1∗ Xiaodong Wang 3∗ Yuxin Guo 1∗ Rui Chen 1∗ Lewei Lu 1 Jifeng Dai 2,4 Yuwen Xiong 2 🖂 {}^{\hrefmailto:[email protected]} 1 Sensetime Research 2 Shanghai Artificial Intelligence Laboratory 3 Peking University 4 Tsinghua University Abstract Generative models have significantly improved the generation and prediction quality on either camera images or LiDAR point clouds for autonomous driving. However, a real-world autonomous driving system uses multiple kinds of input modality, usually cameras and LiDARs, where they contain complementary information for generation, while existing generation methods ignore this crucial feature, resulting in the generated results only covering separate 2D or 3D information. In order to fill the gap in 2D-3D multi-modal joint generation for autonomous driving, in this paper, we propose our framework, HoloDrive , to jointly generate the camera images and LiDAR point clouds. We employ BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models, and introduce a depth prediction branch in the 2D generative model to disambiguate the un-projecting from image space to BEV space, then extend the method to predict the future by adding temporal structure and carefully designed progressive training. Further, we conduct experiments on single frame generation and world model benchmarks, and demonstrate our method leads to significant performance gains over SOTA methods in terms of generation metrics. Figure 1 : Our pipeline HoloDrive can jointly generate realistic street scene video of surround-view cameras and LiDAR point cloud. † † ∗ equal contribution 1 Introduction Generative models have gained significant attention for their ability to understand data distributions and create content, making notable strides in areas such as image [ 31 , 35 , 4 ] and video generation [ 1 ] , 3D object generation [ 48 , 16 ] , and editing [ 13 ] . In the context of simulation, generative models have shown remarkable potential for creating realistic scenarios, which are crucial for training and evaluating safety-critical embodied agents like autonomous vehicles [ 11 , 44 ] . This capability reduces the need for expensive manual modeling of the real world, facilitating extensive closed-loop training and scenario testing. Furthermore, world models are gradually being explored to understand and predict the dynamic nature of the real world, which is crucial for simulation scenes and video generation. Despite the advancements in conditional image and video generation for autonomous driving, existing approaches primarily focus on a single modality, utilizing either 2D data [ 44 , 53 ] or 3D data [ 48 , 52 ] . A truly capable autonomous driving system, however, usually integrates multiple sensors, including both cameras and LiDARs. Cameras provide rich texture and semantic information, while LiDARs offer precise 3D geometric details. The combination of these two modalities enhances perception accuracy, as they are complementary [ 24 , 20 ] . The exploration of joint modality generation is still very preliminary at present. BEVWorld [ 51 ] has made some explorations, but the quality and controllability of generation are still difficult to compare with the SOTA methods in single modality. We propose a holistic 2D-3D generation framework for autonomous driving, HoloDrive , which unifies 2D and 3D generation for street-view autonomous driving data into a single and effective framework. HoloDrive can jointly generate both multi-view camera and LiDAR data, as illustrated in Figure 1 . Our framework extends state-of-the-art 2D and 3D generation models, enabling the generation of realistic street scenes with text and bounding box/map conditions. To achieve joint 2D and 3D generation, we first introduce a depth prediction branch in the 2D generative model, with supervision naturally derived from the 3D LiDAR. We further employ an efficient BEV-to-Camera transformation based on this depth prediction to align the 3D and 2D spaces, and also a Camera-to-BEV module that introduces rich 2D semantic priors to 3D space. These cross-modal structures facilitate effective information exchange between the two modalities during the generation process and making the entire model end-to-end trainable. We apply the joint pipeline on both single frame and video generation tasks with progressive training, combined with extra multi-task learning on the video domain for a smooth transition across training stages. We conduct experiments on the NuScenes dataset [ 3 ] , which provides information including paired multi-view camera images, LiDAR point clouds, text descriptions, and map layouts. Our results show that integrating joint 2D-3D modeling, HoloDrive achieves state-of-the-art performance in generating both single-frame and sequential data of multi-view camera images and LiDAR point clouds. The main contributions of this paper can be summarized as follows: • We present a novel framework, HoloDrive, to jointly generate multi-view camera images and LiDAR point clouds that are consistent in 2D and 3D space, given text and layout conditions. • We propose to add extra depth supervision to the 2D generation and apply an efficient Camera-to-BEV transformation model to align the 2D and 3D spaces, enhancing joint 2D-3D generative modeling, and further extending the joint modeling to video generation. • Our method shows superior generation quality, faithfully following given conditions, as well as 2D-3D consistency, achieving state-of-the-art performance for both single-frame and video generation. 2 Related Work 2.1 Image Generation Image generation is one of the most basic topics in generative modeling and various methods have been explored. Among them, diffusion models, which model the image generation via a reverse iterative stochastic process, raise more and more attention to competitive training stability and generation quality. The reasons behind are carefully design choices in diffusion models, including reducing the prediction resolution by auto-encoder [ 34 ] or cascade model [ 35 ] , better noise scheduler [ 37 , 15 , 28 ] , classifier-free guidance [ 10 ] for control capability and so on. More recently, several works [ 31 , 23 , 27 ] manage to transfer the scaling ability of the Transformer [ 42 ] to diffusion models that has shown priority in the NLP area. Compared with natural images, there exists inherent differences, i.e., regular scene structures and diverse objects in the autonomous driving (AD) area. To compensate for the differences, layout information is utilized to guide generation. For instance, BEVGen [ 38 ] refers to 3D information by projecting all layouts into BEV space. Conversely, BEVControl [ 50 ] begins from projecting 3D coordinates to image views to construct 2D geometric guidance, and MagicDrive [ 6 ] combines the advantage of both methods. Recently, Drive-WM [ 44 ] transfers the pixel-level layout information to latent space and resorts to a unified embedding to attend to them. Our method makes further improvements by introducing point-cloud synergy. 2.2 LiDAR Generation LiDAR point cloud generation has been explored in recent years, a task belonging to 3D point cloud generation. Early works utilized the variational autoencoder (VAE) [ 18 ] or generative adversarial network (GAN) [ 7 ] on point cloud to enable unconditional generation [ 2 , 36 ] . LiDARGen [ 54 ] leverages a score-matching energy-based model and denoise from pure noise into point clouds on the equirectangular view. To better maintain the structure and semantic information of LiDAR scenes, UltraLiDAR [ 48 ] first proposes to utilize a discrete representation to model the distribution

Read the complete text