CrashAgent: Crash Scenario Generation via Multi-modal Reasoning

Miao Li, Wenhao Ding, Haohong Lin, Yiqi Lyu, Yihang Yao, Yuyou Zhang, Ding Zhao · 2025

Read CrashAgent: Crash Scenario Generation via Multi-modal Reasoning by Miao Li, Wenhao Ding, Haohong Lin, Yiqi Lyu, Yihang Yao, Yuyou Zhang, Ding Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18341.txt.

Introduction

Miao Li and co-authors present, in a 2025 paper, a system that turns real traffic crash reports into test scenarios for self-driving cars. Training data brims with ordinary driving, so the dangerous tail of events stays underrepresented. Algorithms cannot learn from crashes they never meet.

Multimodal language models read the reports, pictures and text together. A team of agents converts them into a structured format simulators can run directly. Road layouts and the unfolding of each crash are reconstructed. Human drivers build skill from rare near-misses; fleets cannot wait for the real ones. Rebuilding documented crashes gives simulators a library of the exact moments that matter.

Catalog record

CREATORMiao Li, Wenhao Ding, Haohong Lin, Yiqi Lyu, Yihang Yao, Yuyou Zhang, Ding Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CrashAgent: Crash Scenario Generation via Multi-modal Reasoning Miao Li, Wenhao Ding, Haohong Lin, Yiqi Lyu, Yihang Yao, Yuyou Zhang, Ding Zhao Abstract Training and evaluating autonomous driving algorithms requires a diverse range of scenarios. However, most available datasets predominantly consist of normal driving behaviors demonstrated by human drivers, resulting in a limited number of safety-critical cases. This imbalance, often referred to as a long-tail distribution, restricts the ability of driving algorithms to learn from crucial scenarios involving risk or failure – scenarios that are essential for humans to develop driving skills efficiently. To generate such scenarios, we utilize Multi-modal Large Language Models to convert crash reports of accidents into a structured scenario format, which can be directly executed within simulations. Specifically, we introduce CrashAgent, a multi-agent framework designed to interpret multi-modal real-world traffic crash reports for the generation of both road layouts and the behaviors of the ego vehicle and surrounding traffic participants. We comprehensively evaluate the generated crash scenarios from multiple perspectives, including the accuracy of layout reconstruction, collision rate, and diversity. The resulting high-quality and large-scale crash dataset will be publicly available to support the development of safe driving algorithms in handling safety-critical situations. Keywords: Autonomous Driving, Safety, Multi-modal Large Language Model 1 Introduction Figure 1: CrashAgent the crash report and diagram to scenarios that can be tested in simulators. The evaluation of autonomous vehicles (AVs) heavily relies on the diversity of test scenarios. However, real-world driving scenarios exhibit a long-tail distribution [ 1 ] , where rare and complex events – particularly those critical to safety – occur infrequently but are crucial for performance assessment. Consequently, the generation of safety-critical scenarios [ 2 ] plays a vital role in identifying potential failure modes and enhancing AV safety before deployment. Current approaches to generating safety-critical scenarios, frequently grounded in adversarial attack techniques [ 3 , 4 ] , encounter two main limitations. First, the generated scenarios often lack realism, deviating from plausible or naturalistic driving behavior. Second, these methods tend to explore only a limited portion of the behavioral space, resulting in low scenario diversity. These constraints hinder their effectiveness in capturing the wide and unpredictable range of events that AVs may face in real-world environments. Recent progress in visual language models (VLMs) [ 5 ] presents a promising avenue to address these limitations. Trained on extensive multi-modal datasets, VLMs exhibit strong reasoning abilities that integrate visual and linguistic information [ 6 , 7 ] . These models are capable of interpreting and describing complex interactions between road users and their environments. By utilizing their capacity to understand and reconstruct safety-critical situations, VLMs offer a foundation for generating scenarios that are both diverse and semantically meaningful. However, even state-of-the-art VLMs are still not inherently capable of imagining corner driving cases without external guidance. To facilitate the exploration of infrequent yet critical scenarios, we incorporate real-world crash reports from the National Highway Traffic Safety Administration (NHTSA) as a reference. These reports offer detailed, human-annotated descriptions and structured diagrams of crash incidents, supplying the contextual grounding necessary for VLMs to generate more realistic and diverse scenarios. In this paper, we introduce CrashAgent, a novel multi-agent framework for generating crash scenarios that combines visual reasoning with tool integration (shown in Figure 1 ). CrashAgent is composed of three key components: (1) the Sketch Agent, which produces high-level visual summaries of sketch diagrams using visual tree-of-thought prompting [ 7 ] ; (2) the Road Agent, which translates these summaries into plausible road layouts and traffic contexts; and (3) the Scenario Agent, which transforms the textual summaries into detailed, simulation-ready scripts. Statement of Contributions. This work makes the following key contributions: • We introduce a novel crash scenario generation method that leverages visual tree-of-thought prompting to enhance the reasoning and planning capabilities of VLMs. • We perform a thorough evaluation demonstrating that the scenarios generated by our framework significantly outperform baselines in terms of both diversity and realism. • A large-scale dataset of safety-critical scenarios derived from the NHTSA crash reports is to be released, supporting future research and benchmarking in autonomous driving safety. 2 Related Works Safety-critical and Crash Scenarios Dashboard camera is a common source for recording traffic crash scenarios, and such data has been systematically compiled into video datasets, including MM-AU [ 8 ] and Car Crash Dataset [ 9 ] . However, these datasets are limited to front-view visual data and lack annotations such as road maps and vehicle trajectories, which are essential for training driving algorithms. A promising approach to addressing this limitation involves generating safety-critical scenarios through adversarial perturbations of existing datasets [ 2 , 10 ] , particularly those with comprehensive annotations like nuScenes [ 11 ] , nuPlan [ 12 ] , and the Waymo Open Motion Dataset [ 13 ] . One intuitive strategy is to apply trajectory optimization to maximize collision rate as an objective. For instance, KING [ 3 ] incorporates differentiable vehicle dynamics as constraints to enhance scenario realism. Further improvements are seen in [ 14 ] , which address issues such as implausible diverging paths and unavoidable collisions. FREA [ 15 ] enhances scenario feasibility by integrating the largest feasible region into the optimization process. Another research direction employs Reinforcement Learning (RL) to manipulate vehicle behaviors, attacking the ego vehicle [ 16 , 17 , 18 , 19 ] . For example, [ 19 ] introduces a Prior Risk Estimation Model as a reward function. AdvSim [ 4 ] applies black-box optimization techniques to transform existing scenes into safety-critical scenarios. To enhance scenario diversity, [ 20 ] employs an evolutionary algorithm based on the Quality-Diversity framework [ 21 ] to explore the generation space. Despite their efficiency, adversarial approaches often result in scenarios with constrained realism and limited behavioral diversity. LLM for Scenario Generation LLMs [ 22 ] have demonstrated strong capabilities in following instructions. Consequently, recent studies have increasingly integrated LLMs into scenario generation. For instance, LCTGen [ 23 ] employs GPT to transform scenario description language into vector representations, which are then used to train a conditional generative model for simulating agent behaviors. Building on this approach, [ 24 ] demonstrates that scenarios generated by LCTGen can enhance the safety performance of neural planners. DriveGen [ 25 ] utilizes LLMs in a two-stage pipeline: the first stage generates maps and vehicle assets, and the second employs a VLM to produce trajectories guided by selected waypoints from the initial output. Additionally, LLMs are utilized as constraint generators in simulations. CTG++ [ 26 ] , for example, translates human preferences into cost functions to guide inference within diffusion models. Expanding on this, [ 27 ] incorporates a chain-of-thought mechanism to enhance the cost function generation process. Recent efforts [ 28 , 29 , 30 ] have also investigated controlling generation outcomes through natural language inputs, often in combination with retrieval-augmented generation (RAG). Although most scenarios are generated in vector space, high-fidelity rendering has also been explored. For example, [ 31 ] uses LLMs to convert primitives from retrieved examples into SCENIC [ 32 ] scripts for execution in Carla. Similarly, [ 33 ] enables scenario updates via conversational inputs, which are then rendered in Carla. Despite the integration of RAG,

Read the complete text