PerturbationDrive: A Framework for Perturbation-Based Testing of ADAS Journal: Science of Computer Programming
Hannes Leonhard, Stefano Carlo Lambertenghi, Andrea Stocco · 2026
Read PerturbationDrive: A Framework for Perturbation-Based Testing of ADAS Journal: Science of Computer Programming by Hannes Leonhard, Stefano Carlo Lambertenghi, Andrea Stocco (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23661.txt.
Introduction
Leonhard, Lambertenghi, and Stocco present PerturbationDrive, a test rig for driver assistance systems. Neural driving stacks work in fair weather yet wobble under odd inputs. The framework packs over 30 image tweaks that mimic weather, light, and sensor flaws. Fresh variants shift over time or follow the model's own gaze. Tests run offline on stored sets or live in simulators. Road generation and search tools widen the scene mix further. The setup makes robustness checks repeatable across labs.
Catalog record
| CREATOR | Hannes Leonhard, Stefano Carlo Lambertenghi, Andrea Stocco |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26549 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23661
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PerturbationDrive: A Framework for Perturbation-Based Testing of ADAS Journal: Science of Computer Programming Hannes Leonhard, Stefano Carlo Lambertenghi, Andrea Stocco Abstract Advanced driver assistance systems (ADAS) often rely on deep neural networks to interpret driving images and support vehicle control. Although reliable under nominal conditions, these systems remain vulnerable to input variations and out-of-distribution data, which can lead to unsafe behavior. To this aim, this tool paper presents the architecture and functioning of PerturbationDrive , a testing framework to perform robustness and generalization testing of ADAS. The framework features more than 30 image perturbations from the literature that mimic changes in weather, lighting, or sensor quality and extends them with dynamic and attention-based variants. PerturbationDrive supports both offline evaluation on static datasets and online closed-loop testing in different simulators. Additionally, the framework integrates with procedural road generation and search-based testing, enabling systematic exploration of diverse road topologies combined with image perturbations. Together, these features allow PerturbationDrive to evaluate robustness and generalization capabilities of ADAS across varying scenarios, making it a reproducible and extensible framework for systematic system-level testing. Metadata Table 1 : Code metadata Code metadata description Please fill in this column Current code version v1.0.0 Permanent link to code/repository used for this code version https://github.com/ast-fortiss-tum/perturbation-drive.git Legal Code License MIT Code versioning system used git Software code languages, tools, and services used Python 3.9, Unity (C#), pygame Compilation requirements, operating environments and dependencies Unix (x86/arm) Link to developer documentation/manual https://github.com/ast-fortiss-tum/perturbation-drive/blob/Replication/README.md Support email for questions [email protected] 1 Introduction Advanced driver assistance systems (ADAS) use perception modules to interpret driving environments in real time for tasks such as object detection, segmentation, and control regression [ 43 , 50 , 26 , 15 , 12 , 35 ] . Deep neural networks (DNNs) represent the standard methodology for ADAS perception and currently deliver the best reported performance. Although accurate under nominal conditions, DNNs often fail to generalize to unseen domains. Exhaustive data collection is infeasible, and small shifts in lighting, weather, or viewpoint can cause perception errors [ 26 , 8 , 13 , 19 , 32 , 42 ] that can produce unsafe driving behaviors. Model-level testing provides insights but ignores the closed-loop nature of driving, where perception continuously affects control [ 41 ] . Large-scale validation requires simulators, since real-world testing is unsafe and would demand millions of miles [ 4 , 25 ] . Platforms such as CARLA [ 10 ] , Udacity [ 45 ] , DonkeyCar [ 9 ] , and NVIDIA DriveSim [ 33 ] support reproducible evaluation, but realistic adverse conditions often require custom assets. Perturbation-based methods address this gap by directly manipulating images, ensuring portability across datasets and simulators. In our previous work, we presented PerturbationDrive [ 23 ] , a library that integrates several perturbations from the literature for vision-based ADAS testing [ 17 , 18 , 36 , 22 , 29 , 39 , 2 ] . In this work, we extended PerturbationDrive in several directions. First, we implemented dynamic [ 7 ] and attention-guided variants [ 31 , 21 ] of the original perturbations. Additionally, we integrate perturbations into search-based testing for combined scenario–perturbation exploration [ 30 , 20 , 37 , 38 ] and we added support to the CARLA simulator. PerturbationDrive supports both offline (component-level) [ 34 , 46 , 51 ] and online (system-level) evaluation [ 41 , 27 , 49 , 14 ] to enable systematic and reproducible evaluation of ADAS, covering both robustness and generalization across diverse driving scenarios. 2 The PerturbationDrive Framework 2.1 Objectives The goal of PerturbationDrive is to provide a systematic framework for evaluating ADAS under controlled image perturbations and procedurally generated road scenarios. It consolidates perturbation techniques into a library with standardized configurations for type, intensity, and random seed, ensuring reproducibility and comparability across ADAS models and simulation environments. Beyond offline testing, PerturbationDrive supports closed-loop evaluation in simulators and integrates perturbations with procedural road generation, enabling both robustness benchmarking and generalization analysis in unseen conditions. 2.2 System Architecture The framework comprises three components ( Figure 1 ): the Perturbation Controller , the Simulator Adapter , and the Benchmarking Controller . 2.2.1 Perturbation Controller The Perturbation Controller implements a library of image perturbations available from the literature [ 17 , 18 , 36 , 22 , 29 , 39 , 2 ] . Perturbations are grouped into three categories: i) static perturbations include frame-level modifications such as noise, blur, defocus, weather overlays, geometric distortions, affine transformations, graphic patterns, and color or tone adjustments. ii) dynamic perturbations propose temporal overlays (e.g., rain, snow, smoke, glare) that preserve consistency across frames. iii) attention-guided perturbations : perform targeted distortions applied to salient regions identified by GradCAM or similar methods [ 6 , 40 ] . All perturbations inherit from a common base class that defines the transformation interface for consistency and extensibility. Figure 1 : Overview of the PerturbationDrive framework. Red: Offline execution, Blue: Online Greedy execution, Yellow: Online SBT execution. 2.2.2 Simulator Adapter The Simulator Adapter bridges the perturbation library and simulators. It intercepts raw camera frames, applies perturbations, and forwards modified frames to the ADAS under test. Current implementations support Udacity [ 48 ] , DonkeyCar [ 44 ] , and CARLA [ 10 ] ; additional platforms (e.g., BeamNG [ 11 ] , NVIDIA DriveSim [ 33 ] ) can be added by implementing the adapter interface. The adapter also integrates with procedural road generation to produce diverse topologies and enforces a per-frame processing budget to maintain real-time execution at 30 FPS. 2.2.3 Benchmarking Controller The Benchmarking Controller manages experiments in two main modes: offline and online [ 41 ] . In offline mode, datasets are perturbed, and the resulting model outputs are compared against ground truth or reference predictions. In online mode, the controller supervises closed-loop simulations, injecting perturbations in real time while logging frames, control actions, and vehicle states, and detecting failures such as collisions or lane departures. During online execution, all parameters and execution traces are recorded to ensure reproducibility and enable replay of failure-inducing cases. For both offline and online experiments, perturbations can be applied either manually or through a greedy strategy. Additionally, for online experiments, PerturbationDrive supports a search-based testing (SBT) approach to guide the exploration of perturbations and scenarios. Greedy Testing. Each perturbation is applied to the provided dataset or road scenarios, starting from a low intensity. In offline experiments, all perturbations are evaluated across all intensity levels. In online experiments, if a system failure occurs, the controller proceeds to the next perturbation; otherwise, the perturbation intensity is progressively increased until a failure occurs or all intensity levels have been tested. Search-Based Testing. The combined space of perturbation types, intensities, and road scenarios is too large for exhaustive evaluation. To address this, PerturbationDrive incorporates SBT. Perturbations and scenarios are treated as search parameters, and candidate tests are generated using fitness functions computed from simulation outcomes to prioritize combinations most likely to expose system failures. In our implementation, we leverage OpenSBT [ 38 ] to guide this search process. Treating perturbations as first-class search variables enables unified exploration of environmental diversity and perceptual distortions, moving robustness evaluation beyond ad hoc perturbation studies toward the systematic discovery of safety-critical failures