Self-Supervised Pre-training with Combined Datasets for 3D Perception in Autonomous Driving

Shumin Wang, Zhuoran Yang, Lidian Wang, Zhipeng Tang, Heng Li, Lehan Pan, Sha Zhang, Jie Peng, Jianmin Ji, Yanyong Zhang · 2025

Read Self-Supervised Pre-training with Combined Datasets for 3D Perception in Autonomous Driving by Shumin Wang, Zhuoran Yang, Lidian Wang, Zhipeng Tang, Heng Li, Lehan Pan, Sha Zhang, Jie Peng, Jianmin Ji, Yanyong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12709.txt.

Introduction

Self-driving cars need to read the world in three dimensions, and teaching them takes huge piles of data. Shumin Wang and co-authors present a training method that learns from unlabeled data in this 2025 paper. Work in language and photo AI showed that broad early training pays off. The team asks whether the same holds for the depth sensors used by cars.

Their method pools unlabeled data from several different sources and learns 3D features from scratch. A small adapter part then helps the model adjust to each source, cutting bias from any one data set. The trained model did better on tasks such as finding objects, tracking them, and mapping road space. Gains grew as more training data was added, which hints that still larger pools could help more.

Catalog record

CREATORShumin Wang, Zhuoran Yang, Lidian Wang, Zhipeng Tang, Heng Li, Lehan Pan, Sha Zhang, Jie Peng, Jianmin Ji, Yanyong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Supervised Pre-training with Combined Datasets for 3D Perception in Autonomous Driving Shumin Wang, Zhuoran Yang, Lidian Wang, Zhipeng Tang, Heng Li, Lehan Pan, Sha Zhang, Jie Peng, Jianmin Ji, Yanyong Zhang Abstract The significant achievements of pre-trained models leveraging large volumes of data in the field of NLP and 2D vision inspire us to explore the potential of extensive data pre-training for 3D perception in autonomous driving. Toward this goal, this paper proposes to utilize massive unlabeled data from heterogeneous datasets to pre-train 3D perception models. We introduce a self-supervised pre-training framework that learns effective 3D representations from scratch on unlabeled data, combined with a prompt adapter based domain adaptation strategy to reduce dataset bias. The approach significantly improves model performance on downstream tasks such as 3D object detection, BEV segmentation, 3D object tracking, and occupancy prediction, and shows steady performance increase as the training data volume scales up, demonstrating the potential of continually benefit 3D perception models for autonomous driving. We will release the source code to inspire further investigations in the community. 1 Introduction In recent years, the fields of NLP [ 53 , 1 ] and 2D vision [ 12 , 45 , 16 ] have witnessed tremendous success through pre-training models on a large volume of data. However, in the domain of 3D vision, particularly in the area of 3D perception for autonomous driving, this trend is difficult to achieve due to the lack of properly annotated datasets. Even though many public datasets [ 3 , 11 , 52 , 34 , 4 ] are available, a large portion of the data within them is unlabeled [ 34 , 4 ] because annotating such 3D data is highly complex and time-consuming, rendering annotation far slower than the rate at which data is collected. On the other hand, the data exhibit significant distribution differences [ 67 ] because 3D datasets are often collected independently in different road environments and with different sensor configurations. This poses a challenge for the collaborative use of multiple datasets to further expand the scale of training data. In this work, we attempt to address the scarcity of annotated data by developing a self-supervised pre-training approach on a combination of heterogeneous datasets. Specifically, we focus on multi-modal 3D perception models because public datasets today are collected by different sensors. Furthermore, we note that state-of-the-art 3D perception models on most tasks and benchmarks such as BEVFusion [ 33 ] , CMT [ 72 ] , SparseLIF [ 81 ] and UniTR [ 58 ] are all multimodal models that combine camera images and LiDAR point clouds. Thus, our work focuses on LiDAR-camera fusion-based 3D perception models. Firstly, we extract large amounts of unlabeled data from publicly available datasets including NuScenes [ 3 ] , Lyft [ 11 ] , and ONCE [ 34 ] to pre-train models, focusing on investigating the impact of large-scale unlabeled pre-training data on 3D perception models. To achieve this, we require a self-supervised training method that enables the backbone networks of both image and point cloud modalities to co-evolve from scratch on vast amounts of data. Although previous researches have proposed numerous self-supervised pre-training approaches for images [ 5 , 17 , 65 , 71 ] , point clouds [ 70 , 82 , 19 , 39 ] and aligning the two modalities [ 46 , 85 , 29 , 51 , 26 ] , they often focus either on single modalities [ 5 , 17 , 65 , 71 , 70 , 82 , 19 , 39 ] , cross-modal distillation [ 9 , 85 , 29 ] , or leveraging pre-trained networks [ 9 ] , which does not fully satisfy our requirements. We therefore propose a self-supervised pre-training framework for collaborative learning between point cloud and image data. In this framework, both modalities are unified in the Bird’s Eye View (BEV) perspective, where contrastive learning facilitates mutual enrichment of information across modalities, facilitating collaborative knowledge acquisition from large-scale data. To mitigate potential semantic information loss when elevating images to BEV, we introduce a Masked Autoencoder (MAE) loss for the image modality, ensuring precise semantic capture. Our approach is designed to train backbones of both image and point cloud modalities from scratch, making it well-suited for investigating the effects of large-scale data pre-training on multi-modal 3D perception models. Secondly, we address the domain gap between datasets. 3D data for autonomous driving are typically collected independently using specialized equipment in different road environments, resulting in varied datasets. These datasets may come from different public sources or be privately collected by companies and organizations at various locations, times, and using different equipment. As a result, data from different sources may exhibit significant distributional differences. The distributional bias may lead to training conflicts, resulting in suboptimal outcomes when they are combined for model training. This hinders the further expansion of the training data scale for 3D perception models. To mitigate the impact of domain gap between datasets, we employ a prompt adapter-based training strategy [ 67 ] to disentangle dataset biases from the backbone network. We set learnable prompt parameters for different datasets, which are activated separately during the training process and connected to the backbone network via an adapter. This method allows the model to leverage larger and more diverse data sources more effectively. In this paper, we propose a self-supervised pre-training framework which can leverage heterogeneous unlabeled datasets for 3D perception models in autonomous driving. Our framework is designed to reduce dependency on large annotated datasets and enhance the data scalability in 3D perception domain. The results in four downstream tasks, 3D object detection, 3D object tracking, BEV segmentation, and occupancy prediction demonstrate that our method can effectively improve model performance with heterogeneous unlabeled datasets. Meanwhile, through our pre-training method, the performance of 3D perception models shows a steady improvement as the amount of training data increases, demonstrating the potential to expand the frontier of 3D perception models. To summarize, we make the following contributions: • We propose a self-supervised pre-training method which is able to learn effective 3D representations from scratch on a combination of heterogeneous datasets. • We evaluate our model on four tasks: 3D object detection, 3D object tracking, BEV segmentation and occupancy prediction, and demonstrate the effectiveness of our method. • We scale up the training data volume to 250,000 frames. We observe that the models demonstrate steady performance improvement as the training data volume scales up, which suggests that our method can be further scaled with additional training resources. 2 Preliminary In this section, we present the idea of self-supervised pre-training and categorize the proposed approaches for 3D perception into two groups. Self-supervised pre-training Self-supervised pre-training is a method in which models are trained by constructing pretext tasks, with training labels derived from the data itself. This allows models to learn preliminary knowledge from unlabeled data and converge more efficiently on downstream tasks. In the field of 3D perception, this method is commonly employed to train backbone encoders [ 83 ] , enabling them to acquire accurate and effective feature encoding capabilities, which can be utilized in subsequent downstream tasks. In recent studies on 3D perception , the pretext tasks for self-supervised pre-training

Read the complete text