UAVBench : An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios
Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah · 2025
Read UAVBench : An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios by Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11252.txt.
Introduction
Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah introduce UAVBench, an open benchmark of fifty thousand validated drone flight scenarios for testing autonomous and AI-driven drone systems. Language models increasingly plan missions and make decisions for drones, yet no standard, physically grounded test set existed to grade their reasoning. Each scenario is stored in a structured format covering mission goals, vehicle setup, environment, and numeric risk labels.
A companion set adds fifty thousand multiple-choice questions across ten reasoning styles, from aerodynamics and navigation to ethics and multi-agent coordination. The frame gives machine-checkable scores for drone-specific thinking under realistic conditions. Thirty-two leading language models were evaluated on the benchmark.
Catalog record
| CREATOR | Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11252
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UAVBench : An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios Mohamed Amine Ferrag, Abderrahmane Lakas, and Merouane Debbah Abstract Autonomous aerial systems increasingly rely on large language models (LLMs) for mission planning, perception, and decision-making; yet, the lack of standardized, physically grounded benchmarks limits systematic evaluation of their reasoning capabilities. To address this gap, we introduce UAVBench, an open benchmark dataset comprising 50,000 validated UAV flight scenarios generated through taxonomy-guided LLM prompting and multi-stage safety validation. Each scenario is encoded in a structured JSON schema encompassing mission objectives, vehicle configuration, environmental conditions, and quantitative risk labels, providing a unified representation of UAV operations across diverse domains. Building on this foundation, we present UAVBench_MCQ, a reasoning-oriented extension containing 50,000 multiple-choice questions spanning ten cognitive and ethical reasoning styles—from aerodynamics and navigation to multi-agent coordination and hybrid integrated reasoning. This framework enables interpretable, machine-checkable assessment of UAV-specific cognition under realistic operational contexts. We evaluate 32 state-of-the-art LLMs, including GPT-5, ChatGPT 4o, Gemini 2.5 Flash, DeepSeek V3, Qwen3 235B, and ERNIE 4.5 300B, and find strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making. UAVBench establishes a reproducible, physically grounded foundation for benchmarking agentic AI in autonomous aerial systems and advancing next-generation UAV reasoning intelligence. To support open science and reproducibility, we release the UAVBench dataset (including labeled data), the UAVBench_MCQ benchmark, evaluation scripts, and all related materials on GitHub: https://github.com/maferrag/UAVBench . I Introduction Large Language Models (LLMs) are emerging as powerful tools for enhancing UAV autonomy. Recent studies have increasingly explored integrating LLMs into UAV systems to improve autonomy, decision-making, and communication. Several works demonstrate how LLMs can augment or replace traditional reinforcement learning and optimization frameworks, which often struggle with training complexity and low sample efficiency [ 3 ] . For example, LLMs have been applied to the Internet of Drones via hybrid decision-making frameworks that combine discovery Generation with structured knowledge graphs, allowing interpretable context-aware UAV control [ 9 ] . Other efforts employ LLM-guided reinforcement learning to address security and energy-efficiency trade-offs in heterogeneous UAV networks, achieving improved secrecy rates and robust trajectory optimization [ 20 ] . Similarly, LLM-based in-context learning has been introduced for intelligent data collection scheduling in UAV-assisted networks, outperforming baseline strategies while also revealing vulnerabilities to adversarial manipulation [ 2 ] . Additional work focuses on minimizing the age of information in UAV-assisted sensor networks using an evolutionary-optimization-assisted LLM, demonstrating superior routing efficiency under high node density conditions [ 13 ] . These developments highlight LLMs’ ability to infuse adaptability, interpretability, and semantic reasoning into UAV decision pipelines. Beyond single-agent autonomy, LLMs are also being leveraged for multi-agent UAV coordination and large-scale operational contexts. Recent frameworks utilize iterative structured prompting to optimize multi-hop UAV placements, thereby reducing computational overhead while maintaining near-optimal performance in network backhaul scenarios [ 11 , 7 ] . Other works demonstrate the effectiveness of LLM-based in-context learning for flight resource allocation in wildfire monitoring, where real-time scheduling is critical to minimizing latency and data staleness [ 1 ] . Hierarchical architectures that combine high-altitude platforms and onboard UAV LLMs have been proposed for 3D aerial highway systems, providing both strategic access control and tactical maneuvering [ 15 ] . In the domain of swarm intelligence, LLM-driven role-adaptive frameworks enhance collaboration through semantic communication and dynamic role switching, improving task coverage and generalization in multi-UAV systems [ 12 ] . Likewise, urban trajectory planning approaches merge DRL with LLM reasoning to ensure safe, efficient, and regulation-compliant operations in low-altitude economic airspaces [ 4 ] . Together, these works demonstrate that LLMs are not merely auxiliary tools but are emerging as core enablers of intelligent, interpretable, and scalable UAV autonomy across diverse mission profiles [ 8 ] . Constructing intelligent agents capable of understanding natural language commands and translating them into navigation behaviors remains a central challenge in artificial intelligence [ 19 ] . Although vision-language navigation (VLN) has been extensively studied for ground robots, the aerial domain introduces far greater complexity. UAVs must operate within continuous three-dimensional environments characterized by high degrees of freedom, varying altitudes, dynamic obstacles, and fluid environmental conditions such as wind and lighting changes [ 17 ] . These factors make path planning, spatial reasoning, and language grounding considerably more difficult than in ground-based systems. Moreover, aerial navigation demands fine-grained control over orientation, velocity, and stability, where small errors can lead to mission failure or collisions. Consequently, direct adaptations of ground-based VLN methods—typically optimized for discrete, planar movements—fail to capture the continuous, physics-driven nature of aerial motion and the real-time decision-making constraints inherent to flight [ 10 ] . Despite growing research attention, existing UAV VLN benchmarks and datasets remain limited in terms of realism, task diversity, and physical grounding. Many rely on simplified discrete actions, static environments, or low-fidelity simulators that neglect the continuous control challenges central to UAV operation [ 7 ] . This lack of physical and semantic richness hinders progress toward fully embodied aerial intelligence. Bridging this gap requires specialized platforms and datasets that integrate realistic flight dynamics, multimodal perception, and mission-level reasoning. Such resources would not only enable more accurate simulation of UAV flight behaviors but also facilitate the study of complex reasoning and language-grounded decision-making under real-world constraints. Addressing these limitations motivates the development of unified benchmarks such as UAVBench, which couple scenario-level realism with structured reasoning evaluation, paving the way for end-to-end research on autonomous, language-guided aerial navigation. Unmanned aerial vehicles (UAVs) are increasingly deployed across domains, including disaster response, agriculture, environmental monitoring, traffic observation, and energy infrastructure inspection. However, most missions still depend on human-operated remote control, which is labor-intensive, error-prone, and costly [ 16 ] . Developing autonomous UAV agents that can perceive, reason, and act in complex environments is therefore a critical research objective. Compared to ground-based or indoor agents, UAVs face distinct challenges such as operating in large-scale, dynamic 3D environments, managing costly data collection, and requiring well-defined aerial-embodied tasks. Addressing these challenges requires specialized simulators, datasets, and evaluation frameworks that facilitate training and benchmarking of UAV embodied intelligence [ 5 ] . Beyond task execution, UAV autonomy requires advancing from fine-grained instruction-based navigation to high-level, goal-oriented cognition. Emerging approaches, such as Object Goal Navigation (ObjectNav), demonstrate the potential of semantic-driven navigation, in which agents reach mission-critical targets using abstract goals rather than detailed step-by-step instructions [ 14 ] . Although ObjectNav has been explored in indoor ground settings, its application to outdoor aerial environments remains underdeveloped. At the same time, human-like embodied cognition —processing continuous first-person visual streams for orientation, reasoning, and navigation —is largely absent from current UAV research. Urban airspaces, with their vertical mobility, dynamic obstacles, and dense semantic complexity, present new challenges for autonomous navigation. To advance the field, it is imperative to establish systematic, standardized, and open benchmarks that evaluate the cognition embodied in UAVs and enable robust, scalable autonomy in real-world scenarios [ 18 ] . Our study is guided by the following research questions, designed to investigate how structured, physically grounded UAV scenarios and reasoning-based evaluation frameworks can advance the development of autonomous aerial intelligence: Research Questions • RQ1: How can a unified schema and