Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Driving

Jingzheng Li, Tiancheng Wang, Xingyu Peng, Jiacheng Chen, Zhijun Chen, Bing Li, Xianglong Liu · 2025

Read Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Driving by Jingzheng Li, Tiancheng Wang, Xingyu Peng, Jiacheng Chen, Zhijun Chen, Bing Li, Xianglong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13872.txt.

Introduction

Jingzheng Li and co-authors build a test library for the safety of self-driving systems, arguing that current benchmarks flatter the technology. Popular test sets reward smooth driving on ordinary roads, but they lack the regulated, worst-case scenarios that decide whether a car may ship, and real crashes barely appear in training data. Their library, Safety2Drive, assembles seventy test items drawn from standard safety regulations for direct closed-loop trials.

The library can also manufacture danger: rain, glare, and sensor attacks can be injected across both cameras and laser scanners to see whether the system still copes. Scoring runs on several levels at once, judging the whole driving stack and its parts, such as object and lane detection, separately. The authors present the package as a full path from building scenarios to validating behavior. Their goal is a shared, regulation-shaped yardstick.

Catalog record

CREATORJingzheng Li, Tiancheng Wang, Xingyu Peng, Jiacheng Chen, Zhijun Chen, Bing Li, Xianglong Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Driving Jingzheng Li, Tiancheng Wang, Xingyu Peng, Jiacheng Chen, Zhijun Chen, Bing Li, Xianglong Liu Abstract Autonomous Driving (AD) systems demand the high levels of safety assurance. Despite significant advancements in AD demonstrated on open-source benchmarks like Longest6 and Bench2Drive, existing datasets still lack regulatory-compliant scenario libraries for closed-loop testing to comprehensively evaluate the functional safety of AD. Meanwhile, real-world AD accidents often occur in long-tail edge scenarios, i.e., safety-critical scenario, that are underrepresented in current driving datasets. This scarcity leads to inadequate evaluation of AD performance, posing risks to safety validation and practical deployment. To address these challenges, we propose Safety2Drive, a safety-critical scenario library designed to evaluate AD systems. Safety2Drive offers three key contributions. (1) Safety2Drive comprehensively covers the test items required by standard regulations and contains 70 AD function test items. (2) Safety2Drive supports the safety-critical scenario generalization. It has the ability to inject safety threats such as natural environment corruptions and adversarial attacks cross camera and LiDAR sensors. (3) Safety2Drive supports multi-dimensional evaluation. In addition to the evaluation of AD systems, it also supports the evaluation of various perception tasks, such as object detection and lane detection. Safety2Drive provides a paradigm from scenario construction to validation, establishing a standardized test framework for the safe deployment of AD. 1 Introduction The field of autonomous driving (AD) has witnessed tremendous advancements, with three primary approaches dominating the landscape: modular-based [ 1 ] , end-to-end [ 2 , 3 ] and foundation model-based AD systems [ 4 ] . Traditional modular-based AD systems such as Autoware [ 5 ] and Apollo [ 6 ] are comprised of distinct perception [ 7 ] , planning [ 8 ] , and control modules. The decoupled design between system components may lead to key information loss during transitions and potentially redundant computation as well. In contrast, end-to-end AD systems map raw sensor inputs directly to control signals through deep neural networks. These two AD systems as data-driven paradigms depend on extensive and diverse training data, which can result in shallow semantic comprehension and latent decision errors under complex driving scenarios. Such limitations motivate the exploration of knowledge-aware AD systems that integrate foundation models equipped with scene understanding and reasoning capabilities, as exemplified by recent LLM- and VLM-based AD systems [ 9 , 10 ] . Despite significant progress in AD under natural driving scenarios, real-world accidents predominantly occur in long-tail corner cases, such as jaywalking pedestrians causing planning failures [ 11 ] . These safety-critical scenarios remain insufficiently captured in publicly available driving datasets, leading to incomplete evaluation of AD systems. Specifically, current validation paradigms exhibit two distinct yet complementary limitations. The first paradigm predominantly leverages open-loop benchmarks, e.g., KITTI [ 12 ] , nuScenes [ 13 ] and Waymo. However, as previously analyzed [ 14 ] , these datasets suffer from skewed scenario distribution. For instance, 75% of nuScenes validation set consist of non-interactive, straight-line driving scenarios, highlighting the lack of diverse test cases. The second paradigm employs closed-loop evaluation frameworks such as CARLA on the benchmarks with multi routes, e.g., Town05Long, Longest6 and Leaderboard V2. Such benchmarks predominantly feature over-simplified traffic dynamics, failing to examine driving ability of AD systems under complicated and interactive traffic. To address this limitation, SafeBench [ 15 ] proposes a scenario-centric validation framework for AD safety assessment. However, it only incorporates 8 predefined safety-critical scenarios. Further, Bench2Drive [ 16 ] enhances scenario diversity for training and evaluation, but its testing suite remains restricted to 220 short-duration scenarios. This highlights the critical need for comprehensive test scenario capable of addressing multi-dimensional functional safety requirements. We propose Safety2Drive, a safety-critical benchmark for the evaluation of autonomous driving. Fig. 1 illustrates the overall test framework of the scenario library, which contains three main modules: scenario construction, scenario generalization and scenario evaluation. The scenario construction not only includes a library of hand-designed scenarios but also supports LLM-based scenario generation [ 17 ] as a flexible supplement. Safety2Drive is regulatory-compliant scenario library, which comprises 70 functional test items covering most of standardized scenarios mandated by AD standards and regulations such as ISO 34504. All scenarios are implemented using standardized OpenSCENARIO format with parametric configuration capabilities, allowing dynamic configuration of environmental/traffic parameters for batch testing [ 18 ] . The scenario generalization module supports two safety-critical capabilities that are used to transform standard scenarios into safety-critical scenarios: One is natural environmental corruption including weather-level corruption, sensor-level corruptions and object-level corruptions across camera and LiDAR sensors [ 19 ] . The other is adversarial attack containing three attack vectors including digital attack, physical attack and backdoor attack [ 20 ] . The scenario evaluation module establishes a comprehensive evaluation framework spanning from perception tasks to system-level assessment. It not only supports the evaluation of intelligent perception algorithms but also the evaluation of various AD systems. Figure 1: The pipeline of Safety2Drive. In summary, the proposed Safety2Drive has the following contributions: • Comprehensive standard regulatory functional test coverage. Safety2Drive includes 70 functional test items. To our knowledge, apart from deploying standard regulatory scenarios in real-world, there is currently no open-source scenario library in closed-loop simulation environments that meet standard regulatory requirements. • Safety-critical scenario generalization. Safety2Drive supports scenario generalization, allowing each functional test item to be expanded into various safety-critical scenarios. Specifically, it supports the injection of safety threats such as natural environment corruptions and adversarial attacks, which can be used to assess the robustness and safety of AD. • Evaluation process from perception to system. Safety2Drive not only supports the evaluation of intelligent perception algorithms but also the evaluation of various AD systems, realizing a unified component-to-system evaluation framework. 2 Related Work 2.1 Autonomous Driving Scenario Benchmarks Benchmarking in the field of AD has seen significant advancements, evolving from specialized datasets tailored for specific tasks to more integrated forms capable of evaluating multiple synergistic system components. KITTI [ 12 ] and CODA [ 11 ] focus primarily on perception tasks, while datasets like LanEvil [ 21 ] aims at lane detection and Trajnet++ [ 22 ] is used for trajectory prediction. Further, nuScenes [ 13 ] and Waymo [ 23 ] support the assessment of planning capabilities facilitating the evaluation of various synergistic system components. Although afore-mentioned datasets provide an open-loop assessment, it is difficult to adequately evaluate planning proficiency due to the lack of closed-loop simulation. Thus, Longest6, CARLA Leaderboard V2, Bench2Drive [ 16 ] and ISS-Scenario [ 18 ] provide a closed-loop evaluation with the capability to test AD systems such as Pedestrian crossing , Cut in , and so on. Actually, safety-critical scenario is crucial for testing AD systems [ 24 , 25 ] , but these datasets do not include safety-critical scenario generation capability. Therefore, we propose a benchmark Safety2Drive that not only has comprehensive functional test items required by standard regulations, but also supports safety-critical scenario generation capability including the natural environmental corruption and adversarial attack. Tab. 1 presents a comparison of AD scenario benchmarks. Table 1: Comparison of autonomous driving scenario benchmarks Benchmark Perception Adversarial Attack Environmental Corruption Closed-Loop AD systems Digital Attack Physical Attack Backdoor Attack KITTI [ 12 ] \times \times \times \times \times \times

Read the complete text