An Empirical Evaluation of White-box and Black-box Test Case Prioritization Techniques in CPSs Modeled in Simulink

Aitor Arrieta · 2025

Read An Empirical Evaluation of White-box and Black-box Test Case Prioritization Techniques in CPSs Modeled in Simulink by Aitor Arrieta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10313.txt.

Introduction

Aitor Arrieta compares ways to order software tests for cyber-physical systems built in Simulink, in an empirical study from 2025. These system simulations run long and repeat at several testing levels, so running the most revealing tests first saves real time. Order can be set with code knowledge or without it.

Eleven ordering methods were compared across six models of varied size and complexity, the largest such study in this setting. Code-aware methods edged out blind ones overall. Among code-aware methods, a simple total-greedy rule won on larger models. Blind methods ordered tests faster. The results guide test teams balancing order quality against ordering cost.

Catalog record

CREATORAitor Arrieta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10313

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Evaluation of White-box and Black-box Test Case Prioritization Techniques in CPSs Modeled in Simulink Aitor Arrieta Abstract MATLAB/Simulink is the leading tool for simulating complex Cyber-Physical Systems (CPSs). The simulation models of complex CPSs are typically compute intensive, and the execution of test cases is long. Furthermore, the execution of test cases is typically triggered several times at different “in-the-Loop” test levels (i.e., Model, Software and Hardware-in-the-Loop). Therefore, test optimization techniques, such as test case prioritization, are paramount when testing these systems. In this paper, we present the largest empirical study on test case prioritization techniques for Simulink models by comparing the performance of white-box and black-box test case prioritization techniques. We assess traditional test case prioritization techniques, and we also propose new approaches for use in the context of Simulink models. We empirically compared 11 test case prioritization techniques using six Simulink models of different sizes and complexities. When comparing white-box against black-box test case prioritization techniques, we found that in general, white-box techniques were slightly better than black-box ones. In the context of white-box test case prioritization, the total greedy approach performed better than the additional greedy techniques in larger models. As for the test case prioritization time, black-box techniques were faster, although total greedy techniques were fast enough to be used in practice. K eywords Cyber-Physical Systems ⋅ \cdot Simulink models ⋅ \cdot Tet Prioritization 1 Introduction Simulation-based testing is a commonly used method for testing and verifying complex and untestable systems such as Cyber-Physical Systems (CPSs) [ 14 ] . In CPSs and other complex systems where the software interacts with parallel physical processes, simulation models are employed in early stages before implementing the system [ 65 ] . MATLAB/Simulink is one of the leading tools for modeling and simulating complex dynamic systems in many sectors and domains, including automotive [ 65 , 88 , 92 ] , aerospace [ 20 , 68 ] , railway [ 91 , 84 ] and system of elevators [ 80 , 79 ] . While simulation-based testing has several advantages, the execution of test suites in this context can be lengthy in real industrial settings, with single simulations requiring up to an hour to execute [ 68 ] . To a large extent, this is caused by the complex mathematical models that are required to model the physical parts of such systems. Other reasons include the need for co-simulation to enable high fidelity simulations that rely on realistic set-ups. For instance, Gladisch et al. [ 32 ] reported that for two autonomous driving case studies (adaptive cruise controller and lane keeping case studies), complex vehicle simulation models were required. Their simulations reduced real-time factors down to 10%, meaning that 1 minute of simulation time requires 10 minutes to simulate [ 32 ] . In another recent study [ 34 ] , it was reported that around 8.2 hours were necessary to execute 20 realistic test cases. Besides, such test cases are not executed only once, but several times at different steps and levels, while moving from the simple Model-in-the-Loop (MiL) level to the more complex and realistic simulations that occur at the Hardware-in-the-Loop (HiL) level (see Section 2.2 ). To address the high cost of testing CPSs, test optimization methods have relied on several approaches, including test case selection and test case prioritization techniques. Test case selection techniques attempt to select a subset of test cases from a full test suite that are relevant to a specific testing objective; test case prioritization techniques, in contrast, attempt to place test cases in an order that allows them to achieve their objectives (i.e., fault detection) earlier than might otherwise be possible. Arrieta et al. [ 6 , 7 ] studied how different black-box metrics combined with Pareto-based search algorithms performed when selecting a subset of test cases from a test suite. This problem aimed at reducing as much as possible the test execution time while not compromising test quality. However, when using Simulink models it is also important to detect faults as soon as possible. This way, it is possible the debugging process faster [ 69 ] . When many test cases exist, many test executions can be performed without finding any faults; thus, effective test case prioritization techniques can be useful. A large corpus of studies have proposed and compared different test case prioritization techniques [ 77 , 76 , 78 , 25 , 24 , 26 , 38 , 43 , 44 , 48 , 47 , 59 , 60 , 81 , 53 ] – although these have been conducted outside of the context of CPSs. Henard et al. [ 41 ] compared ten white-box and ten black-box test case prioritization techniques, and concluded that there was little difference between the two classes of techniques in terms of rate of fault detection. However, Henard et al. [ 41 ] focused on unit testing of C programs, which is quite different from testing Simulink models. One major difference is related to semantics. While Simulink models can rely on both discrete and continuous-time semantics, programs written in C often rely only on discrete semantics. This allows the outputs of Simulink models to be connected to their inputs, resulting in continuous feedback over time. On the other hand, coverage-based white-box test case prioritization techniques might not be applicable/effective on some Simulink models due to the difficulty of analyzing coverage for continuous operations [ 66 ] . Furthermore, in the context of Simulink models, test cases are signals that stimulate the inputs of the system over the time. This provides opportunities to provide novel test metrics based on anti-patterns related to signals, which are among those used in this work. In the past, test case prioritization techniques that rely on historical data have performed well, for general purpose software [ 61 , 89 , 73 ] and for simulation-based testing models [ 12 , 11 ] . However, in order for these techniques to be effective, test cases need to be executed several times [ 46 ] . To address this problem, test quality metrics that focus either on black-box techniques (i.e., metrics that measure test quality focusing solely on the inputs and outputs of the simulation models) or on white-box techniques (i.e., metrics that measure which parts of the simulation models have been exercised) can be employed. To the best of our knowledge, this is the first paper in which different white-box and black-box test case prioritization techniques are proposed and empirically evaluated in the context of MATLAB/Simulink models. Specifically, this paper makes the following key contributions. First, we present and revisit test case prioritization techniques for the context of MATLAB/Simulink models. Second, we conduct an empirical evaluation considering six subject models of different sizes and complexities and 11 test case prioritization techniques. We also compared these techniques with a baseline test case prioritization approach and with an optimal test case ordering. To the best of our knowledge, this is the largest test prioritization empirical evaluation performed to date in the context of Simulink models. Third, we provide implementations of all of the techniques that we evaluated in a public repository that can be accessed by other researchers and

Read the complete text