On-Demand Scenario Generation for Testing Automated Driving Systems DOI: 10.1145/3715722 Journal: PACMSE Volume: 2 FSE FSE005 7 CCS: Software and its engineering Software testing and debugging
Songyang Yan, Xiaodong Zhang, Kunkun Hao, Haojie Xin, Yonggang Luo, Jucheng Yang, Ming Fan, Chao Yang, Jun Sun, Zijiang Yang · 2025
Read On-Demand Scenario Generation for Testing Automated Driving Systems DOI: 10.1145/3715722 Journal: PACMSE Volume: 2 FSE FSE005 7 CCS: Software and its engineering Software testing and debugging by Songyang Yan, Xiaodong Zhang, Kunkun Hao, Haojie Xin, Yonggang Luo, Jucheng Yang, Ming Fan, Chao Yang, Jun Sun, Zijiang Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14053.txt.
Introduction
Songyang Yan and nine co-authors generate driving test scenarios on demand at chosen risk levels, in a paper from 2025. Self-driving tests need both natural scenes and near-crash scenes, and most generators favor one extreme. Testing goals vary by region and purpose, so control over risk is essential.
Their framework learns from real traffic data and uses a risk regulator to dial scenario danger up or down. An improved search keeps scenarios diverse across risk levels. Tests in a standard simulator across several driving systems confirm risk control works. Accident types differ by risk level, which proves the dial matters.
Catalog record
| CREATOR | Songyang Yan, Xiaodong Zhang, Kunkun Hao, Haojie Xin, Yonggang Luo, Jucheng Yang, Ming Fan, Chao Yang, Jun Sun, Zijiang Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14053
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-Demand Scenario Generation for Testing Automated Driving Systems DOI: 10.1145/3715722 Journal: PACMSE Volume: 2 FSE FSE005 7 CCS: Software and its engineering Software testing and debugging Songyang Yan, Xiaodong Zhang, Kunkun Hao, Haojie Xin, Yonggang Luo, Jucheng Yang, Ming Fan, Chao Yang, Jun Sun, Zijiang Yang Abstract . The safety and reliability of Automated Driving Systems (ADS) are paramount, necessitating rigorous testing methodologies to uncover potential failures before deployment. Traditional testing approaches often prioritize either natural scenario sampling or safety-critical scenario generation, resulting in overly simplistic or unrealistic hazardous tests. In practice, the demand for natural scenarios (e.g., when evaluating the ADS’s reliability in real-world conditions), critical scenarios (e.g., when evaluating safety in critical situations), or somewhere in between (e.g., when testing the ADS in regions with less civilized drivers) varies depending on the testing objectives. To address this issue, we propose the On-demand Scenario Generation (OSG) Framework, which generates diverse scenarios with varying risk levels. Achieving the goal of OSG is challenging due to the complexity of quantifying the criticalness and naturalness stemming from intricate vehicle-environment interactions, as well as the need to maintain scenario diversity across various risk levels. OSG learns from real-world traffic datasets and employs a Risk Intensity Regulator to quantitatively control the risk level. It also leverages an improved heuristic search method to ensure scenario diversity. We evaluate OSG on the Carla simulators using various ADSs. We verify OSG’s ability to generate scenarios with different risk levels and demonstrate its necessity by comparing accident types across risk levels. With the help of OSG, we are now able to systematically and objectively compare the performance of different ADSs based on different risk levels. † † cc-license: by 1. Introduction Figure 1. Comparison of scenarios at different risk levels. The green vehicle is the ADS under test (i.e., EGO), and the red vehicle is the background vehicle (i.e., NPC). Left scenario: The EGO and NPC are slowly moving in different lanes, with a far distance between them. Middle scenario: a slower NPC attempts to merge into the lane occupied by the EGO, forcing the EGO to yield. Right scenario: An adjacent NPC suddenly changes lanes into the EGO’s lane, resulting in a collision. Ensuring safety is crucial for Automated Driving Systems (ADS). Recent incidents, such as a Tesla in Autopilot mode failing to detect a crossing truck, highlight this need ( Thadani et al. 2023 ) . The National Highway Traffic Safety Administration (NHTSA) has documented 736 crashes involving Tesla vehicles equipped with Autopilot, resulting in 17 fatalities since 2019 ( Siddiqui and Merrill 2023 ) . Testing is vital for safety assurance. It typically includes real-vehicle and simulation tests, with the latter preferred for its cost-effectiveness and zero risk. Testing ADS requires a diverse set of driving scenarios with varying levels of risk. According to a survey of autonomous driving test engineers ( Lou et al. 2022 ) , the common practice involves using data from real-world driving scenarios and accident reports to comprehensively test ADS. However, the testing process faces significant challenges due to the lack of a standardized method for generating scenarios with different risk levels. Firstly, the testing requirements for ADS vary depending on the specific objectives of the tests. In some cases, testers need natural scenarios that closely resemble real-world driving conditions to assess the system’s performance in everyday situations. In other cases, critical scenarios that push the limits of the ADS are necessary to identify potential weaknesses and edge cases. Additionally, scenarios with intermediate levels of risk are crucial for evaluating the system’s behavior in moderately challenging conditions. A generator capable of creating scenarios with various levels of risks and naturalness is essential to meet these diverse testing needs. Figure 1 demonstrates that scenarios with different risk levels possess distinct values. The first scenario, a natural one that frequently occurs in real life, features an NPC (non-player character) vehicle that poses almost no threat to the EGO (ADS under test) vehicle, and maintaining safe driving is the baseline for the ADS. In the second scenario, which has a medium risk level, a slower NPC cuts in closely in front of the EGO, requiring the EGO to yield. The final scenario, characterized by a high-risk level, involves a high-speed cut-in by an NPC, making it nearly impossible for the EGO vehicle to avoid a collision. This scenario can be used to test the ADS’s performance under extreme conditions, assess its capability limits, and determine whether it can minimize losses. Secondly, without a standardized method for measuring the criticalness and naturalness of the scenarios, it is difficult to properly compare the safety of different ADSs, or different versions of a developing ADS. For example, comparing the number of crashes through random testing can be misleading since the randomly generated scenarios may have different levels of naturalness. To address this issue, a consistent and objective way of quantifying the criticalness and naturalness of scenarios is necessary. This would enable testers to generate scenarios with desired characteristics and make fair comparisons during their development of ADS. Recent studies have aimed at automating scenario generation, typically using three methods: data-driven ( Scanlon et al. 2021 ; Abdelfattah et al. 2021 ; Chen et al. 2021 ; Savkin et al. 2021 ) , knowledge-based ( Shiroshita et al. 2020 ; Wang et al. 2021a ; Chang et al. 2024 ; Woodlief et al. 2024 ) , and adversarial generation approaches ( Zhu et al. 2021 ; Ding et al. 2021 ; Ghodsi et al. 2021 ; Arief et al. 2021 ; Koren et al. 2018 ; Cai et al. 2020 ; Wang et al. 2024 ) . Data-driven methods often produce natural but repetitive scenarios due to the data’s long-tail effect. Knowledge-based approaches, relying on human-designed rules, may lack efficiency and struggle with diversity. In contrast, adversarial generation methods, known for their efficiency and flexibility, are increasingly popular. They treat scenario generation as a search problem. However, many of these methods focus either on scenario naturalness or criticalness, failing to offer flexibility in the controlling the two aspects. To generate scenarios that meet the test requirements, we proposed the On-demand Scenario Generation (OSG) Framework, capable of generating scenarios with different risk levels. Grading the risk level of scenarios allows testers to objectively test and compare different ADSs. Testers can select scenarios with different risk levels based on the development stage of the autonomous driving system and specific needs. Generating diverse driving scenarios with varying risk levels presents several challenges. Firstly, quantifying the criticalness and naturalness of scenarios is complex due to the intricate interactions between vehicles and the environment. Secondly, even if there is a way of measuring criticalness and naturalness, we still need to design an efficient searching procedure to avoid generating duplicate scenarios which wastes testing resources and reduce the effectiveness of the testing results. To address the first challenge, we designed a Naturalness Estimator and a Risk Indicator to quantify the naturalness of a scenario and its criticalness. We employ generative models to learn from traffic datasets. These data are gathered from the real world and encompass a broad spectrum of natural traffic behavior patterns. The trained model assesses the naturalness of newly generated