Collaborative Perception Datasets for Autonomous Driving: A Review
Naibang Wang, Deyong Shang, Yan Gong, Xiaoxi Hu, Ziying Song, Lei Yang, Yuhan Huang, Xiaoyu Wang, Jianli Lu · 2025
Read Collaborative Perception Datasets for Autonomous Driving: A Review by Naibang Wang, Deyong Shang, Yan Gong, Xiaoxi Hu, Ziying Song, Lei Yang, Yuhan Huang, Xiaoyu Wang, Jianli Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12696.txt.
Introduction
Naibang Wang and co-authors review datasets for collaborative perception in self driving cars, in a 2025 paper billed as the first of its kind. Cars that share sensor views through vehicle to everything links can see around corners and past blockages. Many datasets support that research, but they differ in sensors, settings, and teamwork style.
The review groups datasets by how the agents cooperate, then compares their data sources, scenes, sensor types, and supported tasks. A detailed side by side analysis runs across several dimensions. Open problems include scaling, diversity, domain shift, standards, and privacy, plus ties to large language models. The authors maintain an online repository that keeps the list current for researchers.
Catalog record
| CREATOR | Naibang Wang, Deyong Shang, Yan Gong, Xiaoxi Hu, Ziying Song, Lei Yang, Yuhan Huang, Xiaoyu Wang, Jianli Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Collaborative Perception Datasets for Autonomous Driving: A Review Naibang Wang, Deyong Shang, Yan Gong, Xiaoxi Hu, Ziying Song, Lei Yang, Yuhan Huang, Xiaoyu Wang, Jianli Lu Abstract Collaborative perception has attracted growing interest from academia and industry due to its potential to enhance perception accuracy, safety, and robustness in autonomous driving through multi-agent information fusion. With the advancement of Vehicle-to-Everything (V2X) communication, numerous collaborative perception datasets have emerged, varying in cooperation paradigms, sensor configurations, data sources, and application scenarios. However, the absence of systematic summarization and comparative analysis hinders effective resource utilization and standardization of model evaluation. As the first comprehensive review focused on collaborative perception datasets, this work reviews and compares existing resources from a multi-dimensional perspective. We categorize datasets based on cooperation paradigms, examine their data sources and scenarios, and analyze sensor modalities and supported tasks. A detailed comparative analysis is conducted across multiple dimensions. We also outline key challenges and future directions, including dataset scalability, diversity, domain adaptation, standardization, privacy, and the integration of large language models. To support ongoing research, we provide a continuously updated online repository of collaborative perception datasets and related literature: https://github.com/frankwnb/Collaborative-Perception-Datasets-for-Autonomous-Driving . I Introduction Perception is fundamental to Autonomous Driving (AD) systems [ 1 ] , [ 2 ] , [ 3 ] , [ 4 ] , [ 5 ] , [ 6 ] , [ 7 ] , [ 8 ] , enabling vehicles to understand their surroundings via onboard sensors for tasks such as object detection [ 9 ] , [ 10 ] , semantic segmentation, tracking [ 11 ] , and trajectory prediction [ 12 ] . The accuracy and robustness of perception directly affect safety and decision-making. Recent advances in computer vision [ 13 ] , [ 14 ] , [ 15 ] , deep learning [ 16 ] , [ 17 ] , and sensor technology [ 18 ] , [ 19 ] , [ 20 ] have led to substantial progress in single-vehicle perception, leveraging cameras [ 21 ] , [ 22 ] , LiDAR [ 23 ] , [ 24 ] , radar [ 25 ] , [ 26 ] , and sensor fusion [ 27 ] , [ 28 ] , [ 29 ] , [ 30 ] , [ 31 ] . However, these systems are constrained by limited field of view, sensor blind spots, and occlusions, making them insufficient in complex, dynamic traffic scenarios. Figure 1: Development Trends of Collaborative Perception Datasets. To address these limitations, Collaborative Perception (CP) leverages Vehicle-to-Everything (V2X) communication to enable real-time data sharing among vehicles and infrastructure [ 32 ] , thereby enhancing the perception of occluded and long-range regions, improving safety, and expanding the Operational Design Domain [ 33 ] , [ 34 ] . For example, at urban intersections, a vehicle may fail to detect occluded agents; V2X links allow perception data exchange with infrastructure or other vehicles to overcome such blind spots. V2X includes Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I), Infrastructure-to-Infrastructure (I2I), and Vehicle-to-Network modes, enabling more comprehensive environmental awareness. High-quality datasets are critical for algorithm development and benchmarking in CP [ 35 ] . Traditional datasets like KITTI [ 36 ] , nuScenes [ 37 ] , OpenMPD [ 38 ] , and Waymo [ 2 ] focus on single-agent perception and lack support for inter-agent communication, making them unsuitable for CP research. Recently, numerous CP datasets, as illustrated in Fig. 1 , have been released to support various scenarios, tasks, and sensor modalities, including DAIR-V2X [ 35 ] , OPV2V [ 39 ] , V2X-Sim [ 40 ] , and V2X-Seq [ 41 ] . These datasets vary in scale, annotation quality, sensor layout, and latency modeling. A systematic review is necessary to clarify their characteristics, limitations, and application scope, and to inform future research directions in collaborative perception. I-A Roadmap Figure 2: Chronological overview of collaborative perception datasets, including roadside, V2V, V2I, V2X, and I2I datasets. Fig. 2 summarizes the chronological evolution of CP datasets across major collaboration paradigms. Development has progressed from static roadside perception to dynamic multi-agent collaboration, driven by increasing task complexity, richer sensors, and advanced communication technologies. Initial datasets, such as Ko-PER [ 42 ] , CityFlow [ 43 ] , and INTERACTION [ 44 ] , focused on roadside perception without inter-agent communication. Around 2020, datasets like T&J [ 45 ] , V2V-Sim [ 46 ] , CoopInf [ 47 ] , and CODD [ 48 ] introduced V2V and V2I collaboration, though often under constrained settings. Later efforts expanded scenario coverage: Adver-City [ 49 ] targets adverse weather, DeepAccident [ 50 ] supports motion and accident prediction, and H-V2X [ 51 ] covers highway scenes. Since 2022, the focus has shifted to real-world deployment and sensor diversity. Datasets such as V2V4Real [ 52 ] , V2X-Real [ 53 ] , and DAIR-V2X-C [ 35 ] emphasize realism, while V2X-Seq [ 41 ] , V2XReID [ 54 ] , and DAIR-V2X-Traj [ 55 ] extend support to trajectory and identity tasks. V2X-Radar [ 56 ] and V2X-R [ 57 ] incorporate 4D radar, and V2XSet-Noise [ 58 ] simulates deployment noise for robustness benchmarking. Recent I2I datasets such as InScope [ 59 ] and RCooper [ 60 ] further diversify collaboration modes. Overall, CP datasets are evolving toward higher realism, richer modalities, and more specialized tasks. I-B Comparison to Related Reviews This review distinguishes itself from prior reviews [ 61 ] , [ 62 ] , [ 63 ] , [ 64 ] , [ 65 ] , [ 66 ] , [ 67 ] , [ 68 ] , [ 69 ] , [ 70 ] , [ 71 ] , which primarily focus on vehicle-centric datasets, specific techniques, or particular fusion strategies. While a few works partially overlap with ours in dataset coverage [ 72 ] , [ 73 ] , they overlook several recent CP datasets and benchmarks, and often lack systematic categorization or comparative depth. In contrast, our review provides a comprehensive and up-to-date compilation of CP datasets, with detailed analysis of their diversity, limitations, and suitability across various CP scenarios. Figure 3: Organization of this paper. I-C Main Contributions The key contributions are summarized as follows: Comprehensive Dataset Review: To the best of our knowledge, this is the first dedicated review that focuses exclusively on CP datasets. We systematically categorize and comprehensively analyze the existing datasets based on collaboration paradigms, data sources, sensor modalities, and application scenarios. Task-Centric Classification: We organize datasets by core perception tasks in AD, including 3D object detection, semantic segmentation, and object tracking and prediction. This task-oriented taxonomy aids researchers in selecting appropriate datasets for specific applications. Recent developments and newly released datasets are also included to ensure timeliness. Comparative Analysis and Future Trends: We conduct multi-dimensional comparisons, highlight representative datasets, and discuss emerging challenges such as standardization, LLM-based annotation, and cross-domain diversity, providing insights for future CP research. I-D Review Structure The organization of this paper is illustrated in Fig. 3 . Section II introduces CP datasets grouped by collaboration paradigms. Section III compares simulated and real-world datasets. Section IV focuses on application scenarios, while Section V analyzes sensor modalities. Section VI covers core perception tasks and