STADA : Specification-based Testing for Autonomous Driving Agents
Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer · 2026
Read STADA : Specification-based Testing for Autonomous Driving Agents by Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10940.txt.
Introduction
Joy Saha and coauthors present STADA, which makes simulation tests for self-driving agents. It builds the tests directly from a formal safety specification written in temporal logic. Given one, it constructs every distinct starting scene. It also builds a diverse set of continuations. It builds runs that reflect the stated behaviors. It does not rely on templates or random scenario building. On several such specs, it more than doubles the best baseline's coverage on the finest criteria. It matches that coverage with far fewer runs.
Catalog record
| CREATOR | Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42598 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10940
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STADA : Specification-based Testing for Autonomous Driving Agents Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer Abstract Simulation-based testing has become a standard approach to validating autonomous driving agents prior to real-world deployment. A high-quality validation campaign will exercise an agent in diverse contexts comprised of varying static environments, e.g., lanes, intersections, signage, and dynamic elements, e.g., vehicles and pedestrians. To achieve this, existing test generation techniques rely on template-based, manually constructed, or random scenario generation. When applied to validate formally specified safety requirements, such methods either require significant human effort or run the risk of missing important behavior related to the requirement. To address this gap, we present STADA , a S pecification-based T est generation framework for A utonomous D riving A gents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic ( LTL f \mathrm{LTL}_{\!f} ). Given a specification, STADA constructs all distinct initial scenes, a diverse space of continuations of those scenes, and simulations that reflect the behaviors of the specification. Evaluation of STADA on a variety of LTL f \mathrm{LTL}_{\!f} specifications formalized in SceneFlow using three complementary coverage criteria demonstrates that STADA yields more than 2 × \times higher coverage than the best baseline on the finest criteria and a 75% increase for the coarsest criteria. Moreover, it matches the coverage of the best baseline with 6 times fewer simulations. While set in the context of autonomous driving, the approach is applicable to other domains with rich simulation environments. ego Lane1 Bike1 inLane@I & inLane@F inLane@I & inLane@F !near_col@I & behind@I !toDRightOf@I & behind@F (a) Relational Graph (RG) (b) Ego and NPC paths (c) Initial Scene (d) Passing Maneuver (e) After Passing \varphi\equiv{\color[rgb]{0,0,1}(behind\_bike\wedge bike\_safe\_distance\wedge\mathbf{F}(in\_front\_bike))}\Rightarrow\mathbf{X}(bike\_safe\_distance\;\mathbf{U}\;in\_front\_bike) Fig. 1 : (a) Relational graph for the precondition of \varphi shown in blue. (b) The ego (red) and NPC (black) paths. (c) Initial scene satisfies behind\_bike and bike\_safe\_distance ( in\_front\_bike pending). (d) During passing, bike\_safe\_distance is violated. (e) After passing, in\_front\_bike is satisfied. The NPC bike is marked with a red arrow and the ego is a blue car. I Introduction The development of autonomous vehicles (AVs) is progressing rapidly with projections that tens of thousands of commercial AVs will be deployed in the coming years [ 1 ] . Like human drivers, these systems must adhere to rules of safe driving, e.g., coming to a complete stop at a stop sign, maintaining a safe following distance, or maintaining a safe distance when overtaking another vehicle. Researchers [ 2 , 3 , 4 , 5 ] have developed promising methods for specifying such properties using pre and postconditions. Here, the precondition defines the context for a driving rule, such as the spatial and temporal relationship of vehicles at an intersection, and the postcondition states allowable AV behavior in that context, like yielding the right-of-way. Validating an AV’s adherence to such properties requires constructing a scenario that meets the precondition. For example, the scenario must show that as the ego vehicle (the vehicle whose behavior is specified by the property) approaches the intersection other vehicles are already there, thereby establishing the context within which the ego vehicle’s behavior can be judged against the postcondition, i.e., whether it yields to the other vehicles or not. Constructing such scenarios is the crux of testing AV agents, and it is typically performed using simulators like CARLA [ 6 ] . Existing methods for performing such simulation-based property validation fall into two categories. Techniques may develop a fixed set of scenarios [ 7 , 8 ] or leverage crash data to reconstruct test scenarios [ 9 , 10 ] to produce tests that meet the precondition. However, there may be multiple ways to establish the precondition, e.g., different timings and numbers of vehicles arriving at an intersection, and this prior work does not explore this diversity. In contrast, fuzzing using algorithmic [ 11 ] or LLM-based approaches [ 12 , 13 ] can produce large, and potentially diverse sets of tests, but those tests may fail to meet the particular precondition rendering them of little value for validating a target property. In this paper, we develop an automated approach to Specification-based Testing for Autonomous Driving Agents ( STADA ). STADA builds on SceneFlow [ 3 ] specifications that describe temporal relationships, using \mathrm{LTL}_{\!f} [ 14 ] , between states of a system that are described by spatial relationships among objects in a scene. Figure 1 shows a SceneFlow specification, \varphi , of an overtaking maneuver of a non-player character (NPC) – a simulation entity not controlled by the agent under test. The precondition (shown in blue) requires that the ego vehicle is behind ( behind\_bike ) and at a safe distance ( bike\_safe\_distance ) from a bike, and in the future it is in front of the bike ( in\_front\_bike ). These atomic propositions are defined using relational logic over a scene graph (SG) computed from sensor inputs, e.g., behind\_bike\equiv ego.aheadOf\cap Bike\not=\emptyset . The test generation challenge is to configure simulation environments that bias the agent controlling the ego to meet this precondition, after which the post-condition can be checked, e.g., that a safe distance is maintained through the maneuver. To achieve this, as depicted in Figure 1 , STADA analyzes the \mathrm{LTL}_{\!f} structure of the specification and defines a set of relational graphs (RG) that contain distinct scenarios that meet the precondition (1.a). These graphs encode requirements both on the initial state in the scenario (1.c) and on the trajectories through the environment established by that initial state. STADA computes the space of trajectories consistent with the precondition (1.b) and selects a diverse subset of those for simulation. The initial state and trajectories are used to generate simulation code that establishes the initial state (1.c) and then biases the ego’s agent towards meeting intermediate (1.d) and final (1.e) states corresponding to the precondition. We implemented STADA using the Scenic [ 15 ] language to generate the initial scene and Python to generate CARLA [ 6 ] simulations. An evaluation using two different driving agents and comparing against three baseline techniques shows that STADA is cost-effective in automating the generation of diverse simulations that are consistent with temporal-logic driving properties. It yields more than twice the coverage of the space of precondition-consistent behaviors than the best baseline technique and can match the coverage of those techniques in 6 times fewer simulations. We describe the STADA approach in Section III , its evaluation in Section IV , and background and related work next. II Related Work and Background We briefly survey related work on coverage, test generation, and property specification and monitoring for AVs and provide relevant background on Scenic and SceneFlow . II-A Coverage and Test Generation for AVs The test adequacy problem aims to determine whether a test suite sufficiently exercises a given system under test (SUT), typically defined through a coverage criteria . Prior work in the AV space has measured coverage for system-level testing based on the number of scenarios of interest that are covered [ 16 ] , where the scenario universe is defined using features such as the road structure [ 17 ] or vehicle interactions [ 18 ] . Specification-based testing