GenAI-based test case generation and execution in SDV platform

Denesa Zyberaj, Lukasz Mazur, Nenad Petrovic, Pankhuri Verma, Pascal Hirmer, Dirk Slama, Xiangwei Cheng, Alois Knoll · 2025

Read GenAI-based test case generation and execution in SDV platform by Denesa Zyberaj, Lukasz Mazur, Nenad Petrovic, Pankhuri Verma, Pascal Hirmer, Dirk Slama, Xiangwei Cheng, Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05112.txt.

Introduction

Writing test cases for software-defined vehicles by hand is slow, and generative artificial intelligence may automate much of it. Denesa Zyberaj, lead author with seven co-authors, present such an approach in a 2025 automotive software paper. Language and vision-language models turn requirements written in natural language, plus system diagrams, into structured Gherkin test cases.

Vehicle signal definitions are standardised through a common specification model, easing integration with testing tools. Generated tests run inside an open, vendor-neutral playground environment. A child presence detection case shows large reductions in manual effort in their evaluation. Human intervention is still needed, a limit the authors state plainly.

Catalog record

CREATORDenesa Zyberaj, Lukasz Mazur, Nenad Petrovic, Pankhuri Verma, Pascal Hirmer, Dirk Slama, Xiangwei Cheng, Alois Knoll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05112

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenAI-based test case generation and execution in SDV platform Denesa Zyberaj, Lukasz Mazur, Nenad Petrovic, Pankhuri Verma, Pascal Hirmer, Dirk Slama, Xiangwei Cheng, Alois Knoll Abstract This paper introduces a GenAI-driven approach for automated test case generation, leveraging Large Language Models and Vision-Language Models to translate natural language requirements and system diagrams into structured Gherkin test cases. The methodology integrates Vehicle Signal Specification modeling to standardize vehicle signal definitions, improve compatibility across automotive subsystems, and streamline integration with third-party testing tools. Generated test cases are executed within the digital.auto playground, an open and vendor-neutral environment designed to facilitate rapid validation of software-defined vehicle functionalities. We evaluate our approach using the Child Presence Detection System use case, demonstrating substantial reductions in manual test specification effort and rapid execution of generated tests. Despite significant automation, the generation of test cases and test scripts still requires manual intervention due to current limitations in the GenAI pipeline and constraints of the digital.auto platform. digital.auto 1 Introduction The transition to software-defined vehicles (SDVs) significantly increases software complexity, intensifying systematic testing challenges. Despite established standards like ASPICE, automotive test planning often remains inconsistent, manual, and poorly aligned with evolving requirements [ 7 , 6 , 4 ] .Requirements expressed in natural language further complicate automated test derivation [ 2 , 1 , 13 ] , limiting traceability [ 5 , 9 ] . Recent advances in Generative AI (GenAI), particularly Large Language Models (LLMs) and Vision-Language Models (VLMs), offer potential solutions by translating ambiguous inputs into structured representations, critical for SDV development. Figure 1: Overview of GenAI-driven test case generation leveraging LLMs, VLMs, and VSS integration for SDV testing automation In this work, as depicted in Fig. 1 , we propose a GenAI-based methodology leveraging LLMs and VLMs to automate the extraction and generation of structured test cases from automotive system requirements provided in natural language and diagrams into executable test cases written in Gherkin 1 1 1 Official Gherkin website: https://cucumber.io/docs/gherkin/ syntax. These are further translated into Python-based test scripts compatible with SDV testing environments . Our methodology integrates Vehicle Signal Specification (VSS), a standardized data schema widely recognized across the automotive industry, to facilitate interoperability across diverse automotive subsystems and standardize the definition of vehicle signals. The entire pipeline is designed to integrate with the digital.auto [ 11 ] playground, a vendor-neutral platform providing digital-twin environments that significantly accelerate the validation of SDV functionalities. We evaluate our approach using the Child Presence Detection System (CPDS), chosen due to its safety-critical nature and event-driven complexity, representing typical challenges faced in SDV testing. The remainder of this paper is structured as follows: Section 2 details our proposed methodology and implementation. Results from our case study on CPDS are presented in Sect. 3 . We conclude with a discussion of our findings and future work in Sect. 4 . 2 Methodology and Implementation 2.1 Methodology Summarization, analysis, and text generation capabilities based on word sequence predictions by LLMs make them suitable for tasks such as VSS signal mapping based on a catalog of VSS signals available in digital.auto, as well as Python code generation based on test case specifications. On the other side, VLMs are leveraged in order to extract relevant information from UML sequence and state diagrams, building upon our work from [ 10 ] . GenAI-driven test case generation consists of the following steps: 1) Signal extraction – VLM interprets the diagram and extracts potential vehicle signals and working process used in communication and control, guided by the prompt; 2) Mapping signals – LLM maps the identified signals from the sequence diagram to the existing VSS format (e.g., Branch, Sensor, Actuator); 3) Gherkin test case generation – visual input (flowchart or state diagram) is used to generate Gherkin test cases relying on multimodal-enabled LLM solution (outside the scope of this paper); 4) Python test file generation – using the refined VSS signals and test logic, the LLM generates an automated Python test file that simulates the test scenarios based on the sequence or state diagram. Figure 2: GenAI-driven test generation workflow leveraging VSS specification and Gherkin test cases. 2.2 Implementation We utilized prompting strategies such as Chain of Thought and Few-Shot Learning. For VSS signal mapping, code generation, and Gherkin test case extraction, we used the LLM GPT-4o. The prompting process was structured into three distinct stages, each explicitly instructing the models to request clarification in case of ambiguities (see Table 1 ). Initial prompts instructed models to generate clearly defined Given-When-Then test scenarios with explicit timing and conditional logic. Subsequent prompts required models to reference only valid VSS signals, explicitly requesting clarification when direct mappings were unavailable. Final-stage prompts directed models to strictly adhere to provided digital.auto playground templates, ensuring compatibility and readability through explicit comments and scenario referencing. Initial outputs were manually validated at each stage to ensure accuracy and realism. This iterative refinement significantly improved reliability. Several technical challenges were identified throughout this process. Notably, ensuring the generated Python test scripts fully match the functional capabilities and constraints of the digital.auto platform, necessitating frequent manual adjustments. Table 1: Prompts overview Task Prompt VSS Signal Extraction from Diagrams You are an expert assistant trained in software architecture and automotive systems. Analyze the uploaded UML state machine diagram, which models the behavior of a component in an automotive software system. Please extract structured information in the format below. Make sure to identify states, transitions, composite (nested) states. Mapping VSS Signals Based on list of identified signals [signal list from diagram] perform mapping with respect to list of possible signals [VSS catalog]. Code generation for digital.auto Generate Python test based on example [digital.auto test example] starting from Gherkin test case [Gherkin test case] taking into account list of available VSS signals [VSS signals]. 3 Use case: Child Presence Detection System A Child Presence Detection System (CPDS) detects and responds to unattended children in parked vehicles. We modeled a simplified CPDS based on Euro NCAP guidelines and relevant patents [ 8 , 12 , 3 ] . CPDS employs direct sensors (cameras, UWB modules, microphones) and indirect sensors (seat pressure monitors, seatbelt indicators), while actuators include HVAC systems, horns, lights, and door locks. Due to this complexity, safety relevance, and its event-driven architecture, the CPDS is ideal for demonstrating structured, scenario-based test case generation using Gherkin [ 14 ] . The CPDS activation cycle initiates immediately after vehicle ignition is switched off (see Fig. 3 ). Within 10 seconds, the system evaluates sensor data to determine if a child is unattended. If detected, the driver is notified and given five minutes to respond. Without a response, CPDS escalates by activating an initial warning. Figure 3: Escalation logic of the CPDS. The system transitions through notification and intervention stages based on time-limited driver responses. Further driver inaction triggers subsequent escalation stages, including activating exterior lights and horn signals, HVAC adjustments, notifying caregivers, unlocking doors, and ultimately contacting emergency services. Afterward, the system returns to standby, awaiting further input or intervention. For demonstration, we focus on modeling and testing the Intervention I – HVAC adjustment step. The structured requirements for this intervention (see Listing 1 ) serve as inputs for our GenAI-driven test case generation. ⬇ Req_CPDS_04 .1.

Read the complete text