Multi-Agent AI Framework for Road Situation Detection and C-ITS Message Generation

Kailin Tong, Selim Solmaz, Kenan Mujkic, Gottfried Allmer, Bo Leng · 2025

Read Multi-Agent AI Framework for Road Situation Detection and C-ITS Message Generation by Kailin Tong, Selim Solmaz, Kenan Mujkic, Gottfried Allmer, Bo Leng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06892.txt.

Introduction

Kailin Tong and four co-authors build a multi-agent AI system that watches road cameras and drafts traffic alerts automatically. Standard detectors perform well in predefined cases but fail on unseen cases and cannot explain them in words. Their system pairs multimodal large language models with vision-based perception, assigning separate agents to detection, distance estimation, decision-making, and message creation for cooperative transport systems.

Evaluated on 103 images drawn from twenty traffic videos, the system achieved perfect recall in situation detection and perfectly formed messages. Weaknesses remained: false-positive detections, and errors in lane counts, lane status, and cause codes. Surprisingly, the newer Gemini 2.5 Flash model performed worse and slower than Gemini 2.0 Flash, motivating specialized fine-tuning for transport applications.

Catalog record

CREATORKailin Tong, Selim Solmaz, Kenan Mujkic, Gottfried Allmer, Bo Leng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06892

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent AI Framework for Road Situation Detection and C-ITS Message Generation Kailin Tong, Selim Solmaz, Kenan Mujkic, Gottfried Allmer, Bo Leng Abstract Conventional road-situation detection methods achieve strong performance in predefined scenarios but fail in unseen cases and lack semantic interpretation, which is crucial for reliable traffic recommendations. This work introduces a multi-agent AI framework that combines multimodal large language models (MLLMs) with vision-based perception for road-situation monitoring. The framework processes camera feeds and coordinates dedicated agents for situation detection, distance estimation, decision-making, and Cooperative Intelligent Transport System (C-ITS) message generation. Evaluation is conducted on a custom dataset of 103 images extracted from 20 videos of the TAD dataset. Both Gemini-2.0-Flash and Gemini-2.5-Flash were evaluated. The results show 100 % 100\% recall in situation detection and perfect message schema correctness; however, both models suffer from false-positive detections and have reduced performance in terms of number of lanes, driving lane status and cause code. Surprisingly, Gemini-2.5-Flash, though more capable in general tasks, underperforms Gemini-2.0-Flash in detection accuracy and semantic understanding and incurs higher latency (Table II). These findings motivate further work on fine-tuning specialized LLMs or MLLMs tailored for intelligent transportation applications. I Introduction Road safety remains a critical challenge in modern mobility. Despite advances in automated driving and Cooperative Intelligent Transport Systems (C-ITS), incidents such as accidents, adverse weather conditions, and hazardous road segments continue to cause significant societal and economic costs. While automated driving technologies promise safer and more efficient transportation, current systems often struggle to react promptly to diverse and unforeseen situations. This limitation underscores the importance of infrastructure-assisted monitoring and safety-warning mechanisms [ 18 , 16 ] . Conventional detection approaches often employ specialized neural networks for accident recognition or anomaly detection. While effective in structured conditions, such methods fail to generalize to complex, open-world scenarios. Recent progress in multi-modal large language models (MLLMs) offers new opportunities for perception and reasoning in Intelligent Transportation Systems (ITS). With strong zero-shot and open-world detection capabilities, MLLMs are well suited for situation detection, where event types are unpredictable and responses must be situationally adaptive [ 3 ] . Yet, MLLMs alone lack domain-specific tuning and robust communication mechanisms. The ESERCOM-D project addresses these challenges by introducing a multi-agent AI framework that combines MLLM-based perception and reasoning with infrastructure sensors for road situation detection and standardized C-ITS message generation. By coordinating agents for detection, distance estimation, decision-making, and Decentralized Environmental Notification Message (DENM) encoding, the framework enables infrastructure to act as an active safety system, enhancing responsiveness and resilience in complex traffic environments. The main contributions of this paper is summarized as follows: • Framework design: We propose a multi-agent AI framework that integrates MLLMs with vision-based perception for road infrastructure monitoring. It generates standardized C-ITS messages in soft real time, with DENM schemas fully compliant with ETSI standards and directly integrable into existing systems. • Empirical insights: Using a real-world camera-image dataset, we benchmark two representative Google MLLMs on detection, information extraction, and decision-making. Results show that while the more advanced model offers broader general capabilities, its added complexity does not necessarily enhance structured ITS tasks and may instead increase latency. • Deployment perspective: We discuss the practical strengths and limitations of the proposed framework, outlining a pathway toward real-world deployment. The remainder of this paper is organized as follows. Section II reviews the related work. Section III introduces the problem statement. Section IV presents our approach in detail. Section VI reports and discusses the evaluation results. Finally, Section VI concludes the paper and provides an outlook on future research. II Related Work Conventional vision-based models for traffic situation detection have achieved strong results in focused subdomains such as accident detection [ 19 ] and road-anomaly detection [ 14 ] . However, their performance often depends on domain-specific training data and they frequently lack robust generalization and explicit scene interpretation beyond closed-world settings. By contrast, MLLMs benefit from extensive pre-training and task-specific finetuning, enabling few- or even zero-shot learning, stronger out-of-distribution handling, and richer contextual reasoning over images and video [ 21 ] . This suggests a natural integration path into ITS for situation analysis and higher-level decision support. In parallel, Vehicle-to-Everything (V2X) communication provides road users and infrastructure with timely, standardized situational information about the environment and near-future routes, thereby augmenting perception and planning capabilities. Motivated by these trends, this work explores the joint use of MLLMs and V2X: we instantiate a practical pipeline for road-situation detection and the generation of standardized response messages for broadcast. In the following, we review the state of the art in both research directions. II- 1 Traffic Situation Detection Using MLLMs Pioneering MLLM-driven efforts for driving scene understanding include DriveGPT4, which proposes an end-to-end autonomous driving and scene reasoning system using MLLM [ 20 ] . Beyond scene description, MLLMs have been used for driver behavior analysis and risk assessment by integrating visual LLMs with reasoning chains [ 22 ] . Liao et al. introduced an MLLM-based grounding architecture that couples text, emotion, image, context, and cross-modal encoders with a multimodal decoder, demonstrating high prediction accuracy and operational efficiency on real-world benchmarks [ 11 ] . Ilhan et al. examined prompt-enhancement strategies for improving interpretability and reported that certain forms of image “enhancement” can act as noise and fail to boost detection performance [ 15 ] . Extending toward decision layers, MLLMs have also been leveraged to diagnose and repair issues in motion planning/trajectory generation pipelines for autonomous vehicles operating in dynamic environments [ 12 ] . Despite this progress, prior work has generally stopped at perception/understanding or diagnosis; relatively few studies couple direct traffic situation detection with standardized response-message generation suitable for V2X broadcast. II- 2 Response-Message Generation An early pipeline proposed by Tong et al. that explicitly connects Connected and Automated Vehicles (CAVs) with large language models for incident detection/interpretation and C-ITS message generation is ConnectGPT [ 17 ] . While the framework shows practical promise, broad validation at scale is limited by dataset size and by the rapid evolution of state-of-the-art MLLMs. Using a different response format, IncidentResponseGPT leverages generative AI to synthesize incident response plans and speed traffic-management recommendations; the authors also discuss risks such as data/guideline bias, ethics of autonomy, and the need for continuous validation [ 8 ] . On the messaging infrastructure side, recent work from RWTH Aachen provides a software stack to use standardized ETSI ITS messages (e.g., CAM/ DENM) within ROS2-based systems and releases a public multi-modal V2X dataset (V2AIX), facilitating research that integrates perception/decision modules with communication stacks [ 9 ] . III Problem Statement Ensuring safe and efficient traffic flow remains a persistent challenge in the deployment of automated and connected vehicles. Prior efforts, such as the EU-H2020 ESRIUM project [ 6 ] , demonstrated the potential of infrastructure-assisted driving by creating precise digital maps of road surface deterioration and using them to guide lane positioning and routing strategies as illustrated in Fig. 1 . While ESRIUM validated these concepts in large-scale on-road demonstrations [ 13 ] , its scope was primarily restricted to road damage monitoring and did not address the dynamic management of unforeseen hazards or the generation of standardized C-ITS messages. Fig. 1: Exemplary scenarios: in-lane offset recommendation (cyan

Read the complete text