AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models

Zijin Zhou, Songan Zhang · 2026

Read AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models by Zijin Zhou, Songan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20878.txt.

Introduction

Zhou and Zhang introduce AITP, a multimodal large language model for assigning blame in traffic accidents. AITP sharpens reasoning with a Multimodal Chain-of-Thought method and adds legal knowledge via Retrieval-Augmented Generation. They also present DecaTARA, a ten-task benchmark that joins ten related traffic accident reasoning tasks with 67,941 annotated videos and 195,821 question-answer pairs. Tests show AITP leads the field across blame calls, crash detection, and crash understanding.

Catalog record

CREATORZijin Zhou, Songan Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32456
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models Zijin Zhou, Songan Zhang Abstract Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Traffic Accident Detection (TAD) and Traffic Accident Understanding (TAU). However, existing studies mainly focus on describing and interpreting accident videos, leaving room for deeper causal reasoning and integration of legal knowledge. Traffic Accident Responsibility Allocation (TARA) is a more challenging task that requires multi-step reasoning grounded in traffic regulations. To address this, we introduce AITP (Artificial Intelligence Traffic Police), a multimodal large language model for responsibility reasoning and allocation. AITP enhances reasoning via a Multimodal Chain-of-Thought (MCoT) mechanism and integrates legal knowledge through Retrieval-Augmented Generation (RAG). We further present DecaTARA , a decathlon-style benchmark unifying ten interrelated traffic accident reasoning tasks with 67,941 annotated videos and 195,821 question–answer pairs. Extensive experiments show that AITP achieves state-of-the-art performance across responsibility allocation, TAD, and TAU tasks, establishing a new paradigm for reasoning-driven multimodal traffic analysis. 1 Introduction Traffic accidents occur in large numbers every year, posing a significant threat to public safety and traffic management 36 . Currently, traffic accident responsibility allocation (TARA) still relies mainly on manual judgment, which is both time-consuming and labor-intensive. Developing an automated TARA system can greatly improve the efficiency of responsibility determination and, to some extent, substitute manual efforts in routine cases. However, research specifically focused on TARA remains largely unexplored. With the rapid development of computer vision and deep learning, earlier studies have achieved automated Traffic Accident Detection (TAD) 7 ; 22 ; 10 ; 11 ; 27 , while recent advances in Multimodal Large Language Models (MLLMs) have shown strong performance in Traffic Accident Understanding (TAU) 38 ; 37 ; 49 ; 29 ; 6 ; 23 ; 25 ; 12 ; 43 ; 21 , particularly in generating textual descriptions of accident scenes. Although existing approaches have achieved remarkable results in both TAD and TAU, they mainly focus on detecting the timing of accidents, identifying the entities involved, and generating textual descriptions of the accident process and its causes. These works have greatly advanced intelligent traffic video understanding but remain at the perceptual and descriptive levels. In contrast, TARA requires a deeper level of reasoning ability—not only to analyze causal relationships and determine responsible parties, but also to comprehend and apply traffic regulations. However, research on enabling multimodal large models to perform such reasoning-driven responsibility determination tasks remains largely unexplored.More importantly,as shown in Fig. 1 , directly applying existing approaches to TARA often leads to hallucination issues, as the models lack structured causal reasoning and external legal knowledge, resulting in unstable or unreliable responsibility judgments. Meanwhile, despite the growing attention on multimodal models for traffic video analysis in recent years, there is still no dataset specifically designed for TARA. Existing datasets 38 ; 37 ; 49 ; 29 ; 6 ; 25 ; 23 mostly focus on accident detection or description tasks, with annotations limited to accident occurrence, categories, and event narratives, lacking systematic labeling of responsibility attribution. Moreover, responsibility determination often involves two participants and complex spatiotemporal interactions, making it challenging to scientifically and objectively annotate responsibility outcomes in traffic accidents. This remains an important yet underexplored problem. Figure 1 : One example to illustrate the limitations of general MLLM in TARA. In the scenario pedestrian was hit while crossing the road. Qwen3-VL fail to detect this issue. In this work, we present DecaTARA (A Decathlon Dataset for Traffic Accident Responsibility Allocation), the first large-scale multimodal dataset specifically designed for TARA. DecaTARA contains X traffic accident videos and Y question–answer pairs. Building upon previous studies in TAD and TAU, we integrate and unify multiple existing datasets to bridge the gap between accident detection and causal reasoning. To comprehensively enhance the model’s capability in traffic accident understanding, DecaTARA is designed with ten tasks, including seven accident-related tasks, two non-accident tasks, and one evaluation task. The accident-related tasks cover accident occurrence judgment, accident type classification, accident fact description, accident reason explanation, accident temporal and spatial location, and avoidance advice. To prevent overfitting and enhance model generalization, we further construct a non-accident subset containing two tasks that describe and infer vehicle behaviors in normal driving scenarios. Finally, we annotate a responsibility allocation subset that quantitatively evaluates models through responsibility allocation accuracy. To the best of our knowledge, DecaTARA is the first large-scale, multi-task dataset dedicated to traffic accident responsibility allocation. Secondly, we propose AITP, a multimodal large language model designed for TARA. AITP introduces two core innovations—Multimodal Chain of Thought (MCoT) 35 ; 45 and Retrieval-Augmented Generation (RAG) 15 — which jointly mitigate hallucination issues during responsibility determination. The model performs step-by-step reasoning on accident occurrence, type, time, subject, fact, cause, and avoidance advice, and ultimately completes responsibility allocation and self-verification. All these sub-tasks are fine-tuned on the DecaTARA dataset to enhance the model’s perceptual and reasoning capabilities, while the responsibility allocation task is accomplished through reasoning based on the previous chain. During this TARA process, the RAG module provides the model with relevant traffic regulation knowledge, enabling fact- and law-grounded reasoning for more reliable and interpretable responsibility attribution. We evaluate our model across multiple tasks in TAD and TAU, and for the first time, introduce responsibility allocation accuracy as a new quantitative metric to more comprehensively assess the model’s understanding and reasoning capability. Experimental results show that AITP achieves state-of-the-art (SOTA) performance on both responsibility allocation accuracy and multi-task benchmarks, significantly outperforming existing baselines. Further ablation studies validate the effectiveness of RAG module, and MCoT reasoning mechanism. Our main contributions are summarized as follows: • We construct DecaTARA ,the first multi-task dataset for traffic accident responsibility allocation, containing 67,941 videos and 195,821 QA pairs, with ten tasks designed to enhance comprehensive accident understanding. • We propose AITP , a multimodal large language model for TARA, which integrates MCoT for stepwise reasoning and RAG for legal knowledge retrieval, effectively reducing hallucination and improving reasoning reliability. • Extensive experiments demonstrate that AITP achieves SOTA performance on both responsibility allocation accuracy and multi-task benchmarks. 2 Related Work Figure 2 : DecaTARA comprises ten tasks: Tasks 1-4 focus on traffic accident detection, Tasks 5-7 are about traffic accident understanding, Tasks 8-9 cover vehicle behavior and cause reasoning in non-accident scenarios, and Task 10 concerns traffic accident responsibility allocation. Traffic Accident Detection and Understanding . Early traffic accident datasets primarily focused on perception-level tasks such as anomaly detection, type classification, and object localization. Representative dashcam datasets, including A3D 41 , DoTA 40 , and CCD 4 , provided temporal and categorical annotations, while DADA 9 and DADA-2000 8 incorporated driver gaze information and fine-grained semantic segmentation. Surveillance datasets 28 ; 39 ; 5 ; 20 ; 38 ; 42 offered broader traffic scenes captured by fixed cameras, and synthetic datasets 1 ; 13 ; 19 ; 32 ; 33 ; 16 ; 17 achieved larger scale and diversity but still suffered from a notable domain gap compared to real-world scenarios. In recent years, research attention has gradually shifted toward TAU, emphasizing multimodal reasoning over accident facts, causes, and prevention. SUTD-TrafficQA 38 pioneered video question-answering for accident understanding, while MM-AU 6 further introduced multi-aspect textual annotations covering factual, causal, and preventive reasoning. TAU-106K 49 expanded

Read the complete text