AgentDrive : An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems
Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah · 2026
Read AgentDrive : An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems by Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16964.txt.
Introduction
Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah build a test set for AI in self-drive cars. Cars with AI must sense, plan, and act with care. Tests for such thought need vast, safe-scene sets. Real crash-risk scenes are too dear to stage. Their 2026 preprint shares AgentDrive, an open set of made scenes.
Big chatbots wrote the scenes in that set. Scenes span hard and risk-full road cases. Teams can train and grade car AI on them. Shared scene sets make grades fair across labs. They may speed safe self-drive work for all.
Catalog record
| CREATOR | Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentDrive : An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah Abstract The rapid advancement of Large Language Models (LLMs) has sparked growing interest in their integration within autonomous systems to enable reasoning-driven perception, planning, and decision-making. However, evaluating and training such agentic AI models remains challenging due to the lack of large-scale, structured, and safety-critical benchmarks. This paper introduces AgentDrive, an open benchmark dataset containing 300,000 LLM-generated driving scenarios designed for the training, fine-tuning, and evaluation of autonomous agents under diverse conditions. AgentDrive formalizes a factorized scenario space across seven orthogonal axes—scenario type, driver behavior, environment, road layout, objective, difficulty, and traffic density—and employs an LLM-driven prompt-to-JSON pipeline to produce semantically rich, simulation-ready specifications that are validated against physical and schema constraints. Each scenario undergoes simulation rollouts, surrogate safety metric computation, and rule-based outcome labeling. To complement simulation-based evaluation, we introduce AgentDrive-MCQ, a 100,000-question reasoning benchmark spanning five reasoning dimensions—physics, policy, hybrid, scenario, and comparative—to systematically assess the cognitive and ethical reasoning of LLM-based agents. We conducted a large-scale evaluation of fifty leading LLMs on the AgentDrive-MCQ benchmark to measure their reasoning capabilities across these five dimensions, covering models such as GPT-5, ChatGPT 4o, Gemini 2.5 Flash, DeepSeek V3, Qwen3 235B, ERNIE 4.5 300B, Grok 4, Mistral Medium 3.1, and Phi 4 Reasoning Plus. Results reveal that while proprietary frontier models dominate in contextual and policy reasoning, advanced open models are rapidly closing the gap in structured and physics-grounded reasoning. To support open science and reproducibility, we release the AgentDrive dataset (including labeled data), the AgentDrive-MCQ benchmark, evaluation scripts, and all related materials on GitHub: https://github.com/maferrag/AgentDrive . I Introduction The rise of large language models (LLMs) has sparked broad interest in applying them to autonomous driving (AD) due to their strong reasoning and conversational abilities [ 22 ] . Researchers have introduced the concept of LLM4AD, designing AD systems that leverage LLMs for tasks ranging from perception and scene understanding to decision-making [ 7 , 6 , 9 ] . For example, Cui et al. [ 7 ] proposed an LLM4AD framework with a comprehensive simulation benchmark to evaluate how well LLMs follow driving instructions. Initial experiments in both simulation and real-world driving showed that LLMs can indeed enhance an autonomous vehicle’s understanding of complex environments and improve human–vehicle interactions. These findings, echoed by recent surveys [ 22 , 6 ] , suggest that combining LLMs with vision models could enable more open-world perception, logical reasoning, and adaptive learning than traditional rule-based or end-to-end systems. At the same time, the LLM4AD concept paper and surveys emphasize that this nascent field faces significant challenges, such as ensuring real-time performance and safety, which must be addressed as development continues. Integrating LLMs into an autonomous driving agent raises new complexities in how the vehicle interprets instructions, interacts with humans, and adheres to traffic rules. One approach to this is DriVLMe, an LLM-based driving agent augmented with both embodied experiences (learned via a driving simulator) and social experiences from real human dialogues [ 10 ] . The goal of DriVLMe is to facilitate natural communication between humans and self-driving cars and to handle long-horizon navigation tasks through free-form dialogue. DriVLMe demonstrated competitive performance in open-loop simulations and closed-loop user studies; however, it also revealed several limitations, including unacceptably high inference times, imbalanced training data, and difficulty handling sudden environmental changes. To ensure LLM-driven vehicles make decisions that are not only effective but also legally and ethically aligned, other works focus on injecting domain knowledge and safeguards into the decision process. For example, Cai et al. [ 2 ] present a retrieval-augmented reasoning framework where a Traffic Regulation Retrieval module automatically fetches relevant traffic laws and safety guidelines based on the vehicle’s situation, and an LLM then interprets these rules to assess each action’s legality and safety. This approach yields an interpretable decision-making pipeline that adapts to regional regulations and provides transparency into why the AI chooses a certain action. Similarly, Kong et al. [ 12 ] introduce a multi-agent superalignment framework to enforce data security and policy compliance in LLM-driven cars. Their system safeguards sensitive vehicle data (e.g., precise locations, camera feeds) from potential leaks and filters the LLM’s queries and outputs, ensuring that driving commands do not violate safety rules or human values. These efforts demonstrate how researchers are enhancing LLM-based driving agents with embodied learning and alignment mechanisms, enabling autonomous vehicles to interact naturally while adhering to legal, safety, and trust constraints. Beyond high-level dialogue and compliance, LLMs and multimodal models are being used to tackle core perception and planning challenges in autonomous driving. Bai et al. [ 1 ] argue that prior vision–language planning approaches, which tokenize only 2D images, cannot reliably perceive a 3D world. They propose a planner that uses DETR-style 3D perceptrons as tokenizers, feeding the LLM with 3D object queries from multi-view camera images. This strategy provided rich geometric context, yielding superior performance in 3D object detection and end-to-end planning on the nuScenes dataset, suggesting that 3D-tokenized LLMs could be key to more reliable path planning. Sah et al. [ 17 ] integrate deep learning models for traffic sign recognition and lane detection with a lightweight multimodal LLM that can reason about the scene. This hybrid system achieved near-perfect accuracy in clear conditions and maintained robustness under adverse weather and occlusions, demonstrating the LLM’s ability to interpret contextual clues. Other studies leverage LLMs for specialized tasks such as parking in mixed traffic [ 11 ] and interpretable end-to-end control. Chen et al. [ 4 ] fuse object-level vector modalities with LLM reasoning to provide explainable driving decisions, while Xu et al. [ 21 ] propose DriveGPT4, which generates both low-level control signals and natural language explanations for each action. These advances highlight LLMs as powerful engines for enhancing perception, planning, and transparency in AD systems. Fig. 1 : Overview of the AgentDrive benchmark suite, which comprises three complementary datasets: AgentDrive-Gen for LLM-based driving scenario generation, AgentDrive-Sim for simulation and outcome labeling, and AgentDrive-MCQ for reasoning and decision-making evaluation. To validate and further develop LLM-driven autonomous vehicles, the community is creating new benchmarks and evaluation tools to enhance their development. One notable effort is Bench2ADVLM, a closed-loop evaluation framework introduced by Zhang et al. [ 24 ] to test vision–language AD models in interactive settings. Bench2ADVLM enables real-time closed-loop testing across simulators and physical vehicles, with a scenario generator that uncovers failure modes. Results indicate that current systems still exhibit limitations under interactive conditions. In parallel, MAPLM [ 3 ] provides a large-scale multimodal dataset combining maps, LiDAR, panoramic images, and Q&A annotations, highlighting the importance of domain-specific data for grounding LLMs in driving contexts. Gao et al. [ 9 ] survey how foundation models can generate and analyze diverse scenarios, showing their potential for evaluating and handling safety-critical cases. Despite progress, challenges persist in latency, interpretability, privacy, regulatory compliance, and robustness. Addressing these motivates the creation of AgentDrive , an open benchmark that exposes LLM-based agents to diverse, safety-critical, and long-horizon scenarios through systematic closed-loop evaluation. Building on the challenges of scalable and realistic scenario generation, our work poses two key research questions