LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems

Venkata Sai Aswath Duvvuru and Bohan Zhang, Michael Vierhauser, Ankit Agrawal · 2025

Read LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems by Venkata Sai Aswath Duvvuru and Bohan Zhang, Michael Vierhauser, Ankit Agrawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11864.txt.

Introduction

Venkata Duvvuru and three co-authors present a testing framework for small drones in a 2025 paper. Small uncrewed aircraft must be checked in simulation across weather, terrain, and mission types before they fly. Today that work is mostly manual: people invent scenarios, set up simulators, plan missions, and sift results by hand.

Their framework, AutoSimTest, hands those steps to a team of large language model agents. The agents create scenarios, prepare the test setup, generate missions, and offer an interactive way to explore results. The design adapts to different drone uses and simulators. The team tested it on two widely used flight controllers, PX4 and ArduPilot, in simulation.

Catalog record

CREATORVenkata Sai Aswath Duvvuru and Bohan Zhang, Michael Vierhauser, Ankit Agrawal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems Venkata Sai Aswath Duvvuru and Bohan Zhang, Michael Vierhauser, Ankit Agrawal Abstract Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose AutoSimTest , a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that AutoSimTest significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort. I Introduction Simulation testing is a critical step in the small Uncrewed Aerial Systems (sUAS) development process, to validate the behavior of sUAS across a variety of real-world scenarios [ 12 ] . This includes, for example, adverse weather conditions (e.g., wind, rain, or fog), diverse terrains (e.g., hilly, flat, urban, and open fields), and varying mission profiles (e.g., long-range, short-range, surveillance, and tracking) [ 28 , 45 , 16 ] . Despite the availability of various sUAS simulation tools and software applications [ 52 , 19 ] , simulation testing remains predominantly a manual, or at best partially-automated process. As a result, current sUAS simulation testing practices result in three main pain points for sUAS application developers. First, identifying and designing simulation test scenarios that clearly specify (a) operational contexts , including weather conditions, environmental settings, terrain variability, (b) sUAS sensor specifications including how noisy each sensor is expected in the scenario, (c) mission objectives , including higher-order tasks, sUAS mission modes, and (d) core safety properties to test is a challenging and time-consuming task. Due to the fact that sUAS operate in complex environments and are expected to perform reliably in unique or even unimaginable real-world scenarios [ 35 , 65 ] , developers’ domain knowledge of sUAS use cases plays an important role. An example, where a software issue caused an incident, is a recent case of a drone crashing in the woods while autonomously tracking a person on an electric scooter. The individual was moving at a moderate speed when suddenly, a dog crossed their path. The drone mistakenly switched its tracking from the person to the dog and crashed into a tree. Video of this drone crash is available on Reddit 1 1 1 https://www.reddit.com/r/dji/comments/1amf9rg/oops_solid_crash_while_tracking_myself_on_the/ . Such unique scenarios are particularly difficult for sUAS developers to imagine and accurately test during the development phase. Second, sUAS developers must model each identified scenario to align with the System under Test (SuT) requirements, and configure the simulation tool to simulate environmental context. This process is often manual, repetitive, time-consuming, and prone to human error. Third, once simulation tests are executed, analyzing flight logs to diagnose abnormal behavior is labor-intensive, requiring a deep understanding of thousands of flight controller parameters [ 15 ] . While tools like PX4 Flight Review [ 62 ] plot sensor data automatically, developers still need to interpret the plots and identify issues manually, which requires specific skills and in-depth knowledge of the flight controller. Furthermore, there is no automation framework that enables sUAS developers to analyze simulation artifacts automatically. Fig. 1 : Overview of our AutoSimTest Framework with the 3 main Phases: Scenario Blueprint Construction , Manual Validation and Feedback (blue), Scenario Specification, Validation, and Execution (green), and Scenario Analysis (yellow). These challenges highlight the need for automated tools and supporting frameworks capable of systematically specifying , generating , executing , and analyzing diverse simulation tests to achieve scalability and sufficient simulation test coverage. Therefore, in this paper, we present a novel framework called AutoSimTest as a first step towards automating the simulation testing process for sUAS. We leverage our experience in simulation and field testing within this domain to construct and implement this framework employing recent advancements in Generative AI, more specifically Large Language Models (LLMs) to (a) generate unique test scenarios that incorporate environmental context, mission objectives, and core safety properties to test, (b) model sUAS mission in SuT expected language, (c) configure the simulation environment based on the test scenario’s environmental context, and (d) produce a simulation report based on the collected data from scenario execution, as well as provide interactive methods for analyzing simulation results for sUAS developers. The contributions of this paper are as follows: • AutoSimTest : This is the first work to propose a novel Multi LLM-Agents-based framework to automate the sUAS simulation testing process. • Automated Flight Analysis : As part of the framework, we introduce a novel Analytics-Agent enabling automated and interactive analysis of sUAS flight logs. • Performance and Developers’ Perception: Extensive experiments have demonstrated that AutoSimTest is capable of testing diverse SuT, effectively analyzing flight logs, and providing valuable support to both novice and experienced developers in better interpreting the simulation data. The remainder of the paper is laid out as follows. In Section II provide a brief introduction to relevant LLM techniques and then, in Section III introduce our AutoSimTest framework. Subsequently, in Section IV and Section V we discuss the different LLM agents part of our framework. We then, in Section VI , describe our evaluation setup for addressing feasibility, generalizability, and agent performance, and report on results in Section VII . Finally, we discuss several lessons learned in Section VIII , threats to validity in Section IX related work in Section X , and conclude in Section XI . II Background This section briefly introduces the LLM techniques used in the design of our AutoSimTest framework (cf. Section III ). II-A Retrieval Augmented Generation (RAG) RAG is a method that combines the strengths of pre-trained language models with external knowledge retrieval. Instead of solely relying on the internal knowledge of the language model, RAG retrieves relevant information from external data sources (e.g., a vector database) – serving as a “knowledge library” that the generative AI models can understand. This approach allows the agents to utilize contextually relevant information, especially in specialized domains such as sUAS [ 49 , 78 ] to generate text. RAG consists

Read the complete text