SynSHRP2: A Synthetic Multimodal Benchmark for Driving Safety-critical Events Derived from Real-world Driving Data
Liang Shi, Boyu Jiang, Zhenyuan Yuan, Miguel A. Perez, Feng Guo · 2025
Read SynSHRP2: A Synthetic Multimodal Benchmark for Driving Safety-critical Events Derived from Real-world Driving Data by Liang Shi, Boyu Jiang, Zhenyuan Yuan, Miguel A. Perez, Feng Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06276.txt.
Introduction
Liang Shi and co-authors build a public fake data set of crashes and near-crashes for driving research. Real crash data is rare and bound by privacy rules, so few teams can use it. The source is the largest natural driving study to date, with thousands of such events. Their set holds over 1,800 crashes and 6,900 near-crashes drawn from that study.
Faces and plates were replaced by AI-made keyframes that keep the safety facts of each scene. Notes mark event type, road and traffic states, and motion data for five seconds around the event. Synced text tales aid scene reading by machines. Two benchmark tasks show how the set can test event sorting and scene sense. The authors aim to open safety work to more labs.
Catalog record
| CREATOR | Liang Shi, Boyu Jiang, Zhenyuan Yuan, Miguel A. Perez, Feng Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SynSHRP2: A Synthetic Multimodal Benchmark for Driving Safety-critical Events Derived from Real-world Driving Data Liang Shi, Boyu Jiang, Zhenyuan Yuan, Miguel A. Perez, Feng Guo Abstract Driving-related safety-critical events (SCEs), including crashes and near-crashes, provide essential insights for the development and safety evaluation of automated driving systems. However, two major challenges limit their accessibility: the rarity of SCEs and the presence of sensitive privacy information in the data. The Second Strategic Highway Research Program (SHRP 2) Naturalistic Driving Study (NDS)—the largest NDS to date—collected millions of hours of multimodal, high-resolution, high-frequency driving data from thousands of participants, capturing thousands of SCEs. While this dataset is invaluable for safety research, privacy concerns and data use restrictions significantly limit public access to the raw data. To address these challenges, we introduce SynSHRP2, a publicly available, synthetic, multimodal driving dataset containing over 1,874 crashes and 6,924 near-crashes derived from the SHRP 2 NDS. The dataset features de-identified keyframes generated using Stable Diffusion and ControlNet, ensuring the preservation of critical safety-related information while eliminating personally identifiable data. Additionally, SynSHRP2 includes detailed annotations on SCE type, environmental and traffic conditions, and time-series kinematic data spanning 5 seconds before and during each event. Synchronized keyframes and narrative descriptions further enhance its usability. This paper presents two benchmarks for event attribute classification and scene understanding, demonstrating the potential applications of SynSHRP2 in advancing safety research and automated driving system development. 1 Introduction Driving safety research is increasingly using approaches based on artificial intelligence to tackle tasks such as crash detection, scene understanding, driver monitoring, and unsafe maneuver detection. These advancements rely on high-quality multimodal datasets with accurately labeled safety-critical events (SCEs), including crashes and near-crashes. However, the rarity of such events, high collection costs, lack of data labels, and strict privacy regulations make accessing suitable datasets challenging, hindering progress and fair benchmarking across models. A common limitation for publicly accessible transportation safety data has been a lack of SCEs due their rarity Yu et al. 2020 ; Geiger et al. 2013 ; Caesar et al. 2020 ; Jain et al. 2016 . The average police-reported crash rate in the US is 3.29 crashes per million miles traveled in 2022, and thousands of hours of data collection are needed to capture a single crash Scanlon et al. 2024 . The majority of the publicly accessible data is not large enough to capture a reasonable number of crashes for robust analysis. For instance, the popular BDD100k dataset contains over 1,000 hours of driving videos with time-series data but lacks labeled SCEs Yu et al. 2020 . Large-scale naturalistic driving study (NDS) could be a valuable source to address the rarity of SCEs. NDS is characterized by continuous driving data collection using multiple sensors instrumented on participants’ vehicles under natural driving conditions. The largest NDS to date, the Second Strategic Highway Research Program (SHRP 2) contains millions of hours of driving data with thousands of SCEs identified Dingus et al. 2016 . The SHRP 2 NDS dataset required investing approximately $155 million in data collection, storage, and management, resulting in over 1,000,000 hours of driving data. The collected data include four camera views, 3-D acceleration, radar, yaw rate, GPS, and lighting Federal Highway Administration 2017 . High-quality annotation is another major challenge. Driving scenarios are complex and involve environmental factors, traffic flow conditions, traffic control, the ego vehicle, and driver behavior. Annotating and labeling SCEs requires expertise of the safety domain and considerable resources. For example, the SHRP 2 NDS undertook a comprehensive research project to develop effective annotation methodologies for its extensive dataset. Factors directly related to driving and how to operationally annotate a driving scene is a huge undertaking Hankey et al. 2016 . However, due to strict privacy policies, access to such data requires significant effort and costs Perez et al. 2016 . As with SHRP 2, researchers must undergo rigorous user certification processes, including Institutional Review Board approval, and are granted only time-limited access under strict data use policies. Artificially generated synthetic data, produced by platforms such as Wayve’s GAIA-1 Hu et al. 2023 and the CARLA simulator Dosovitskiy et al. 2017 , offers an alternative source of multimodal driving data. However, challenges persist with synthetic data in SCE-related research: accurately configuring the numerous parameters influencing SCEs is complex; ensuring high fidelity to real-world conditions is difficult; and synthetic datasets may not fully capture the variability of actual driving scenarios, limiting their practical applicability. Another challenge in publicly accessible multimodal driving data is the lack of benchmarking Arvin et al. 2021 ; Shi et al. 2022 ; Taccari et al. 2018 ; Winlaw et al. 2019 ; Osman et al. 2019 ; Peng et al. 2020 ; Chan et al. 2017 ; Taccari et al. 2018 ; Le et al. 2020 ; Bao et al. 2020 ; Simoncini et al. 2022 ; Radu et al. 2021 ; Yao et al. 2019 . For instance, both Shi et al. 2022 and Arvin et al. 2021 utilize the SHRP 2 NDS dataset for crash detection algorithm development but with different configurations. Shi et al. 2022 uses 59,997 normal driving instances, 1,820 crashes, and 6,848 near-crashes for three-way classification (crash vs. near-crash vs. normal driving), whereas Arvin et al. 2021 selects a smaller subset consisting of 7,566 normal driving instances and 1,315 crashes and near-crashes for two-way classification (crash/near-crash vs. normal driving). This variability in data usage, combined with differences in implementation and hyperparameter tuning, makes it challenging to compare results fairly. These challenges highlight the urgent need for standardized datasets and evaluation protocols to enable consistent and fair benchmarking in driving safety studies. This study advances driving safety assessment by introducing a fully public driving safety evaluation dataset and establishing a benchmark for SCE risk evaluation, including attribute detection and scene understanding. The dataset integrates multimodal data—tabular records, time-series signals, keyframe images, and natural language descriptions—to support research in crash prediction, driving behavior analysis, and multimodal learning. To balance privacy and data utility, we develop a Stable Diffusion-based workflow with ControlNet to de-identify personally identifiable information (PII) while preserving critical driving context. Derived from the SHRP 2 NDS, the dataset contains 1,874 crashes and 6,924 near-crashes, each labeled with event type, conflict type, and incident type. It also includes five key timestamps, time-series sensor data spanning 5 seconds before and during each event, and annotated narrative descriptions, providing a comprehensive resource for benchmarking event attribute classification and scene understanding in driving safety research. The rest of the paper is organized as follows: related works are discussed in Section 2; Section 3 and 4 detail the SynSHRP2 data and processing workflow; Section 5 presents two tasks along with their corresponding benchmarks; and summary and conclusion are provided in Section 6. 2 Related Works 2.1 Publicly accessible multimodal driving datasets There are two main classes of multimodal driving datasets: real-world and synthetic. Real-world datasets include nuScenes, which offers a comprehensive, multimodal sensor suite for complex urban scenarios Caesar et al. 2020 ; the Waymo Open Dataset, which provides synchronized multi-sensor data—incorporating LiDAR, radar, and multiple cameras—along with detailed 3D object annotations Sun et al. 2020 ; and KITTI, a benchmark that has set standards for object detection and tracking Geiger