Fuzzing-based Mutation Testing of C/C++ CPS

Jaekwon Lee, Fabrizio Pastore, Lionel Briand · 2025

Read Fuzzing-based Mutation Testing of C/C++ CPS by Jaekwon Lee, Fabrizio Pastore, Lionel Briand (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24100.txt.

Introduction

Jaekwon Lee, Fabrizio Pastore, and Lionel Briand test faulty-code hunting in software for cyber-physical systems, in a 2025 study. Mutation testing plants small code changes and checks whether test suites catch them. It is advised for safety-critical embedded code, often written in C or C++. Standard tools rely on symbolic execution, which stalls on black-box parts and other hard cases.

The team instead uses fuzz testing, which feeds random inputs to explore program paths. On parts from working satellite systems, their method caught forty to ninety percent of the mutants that developer tests missed. It beat symbolic execution by up to fifty points. Mixing both methods added less than one point, so fuzzing alone carried the gains.

Catalog record

CREATORJaekwon Lee, Fabrizio Pastore, Lionel Briand
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-24100

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fuzzing-based Mutation Testing of C/C++ CPS Jaekwon Lee, Fabrizio Pastore, Lionel Briand Abstract Mutation testing can help minimize the delivery of faulty software. Therefore, it is a recommended practice for developing embedded software in safety-critical cyber-physical systems (CPS). However, state-of-the-art mutation testing techniques for C and C++ software, which are common languages for CPS, depend on symbolic execution. Unfortunately, symbolic execution’s limitations hinder its applicability (e.g., systems with black-box components). We propose relying on fuzz testing, which has demonstrated its effectiveness for C and C++ software. Fuzz testing tools automatically create test inputs that explore program branches in various ways, exercising statements in different program states, and thus enabling the detection of mutants, which is our objective. We empirically evaluated our approach using software components from operational satellite systems. Our assessment shows that our approach can detect between 40% and 90% of the mutants not detected by developers’ test suites. Further, we empirically determined that the best results are obtained by integrating the Clang compiler, a memory address sanitizer, and relying on laf-intel instrumentation to collect coverage and guide fuzzing. Our approach detects a significantly higher percentage of live mutants compared to symbolic execution, with an increase of up to 50 percentage points; further, we observed that although the combination of fuzzing and symbolic execution leads to additional mutants being killed, the benefits are minimal (a gain of less than one percentage point). ∎ 1 Introduction Ensuring high-quality test suites is essential for quality assurance of embedded software in cyber-physical systems (CPS); indeed, this is the reason why independent software validation and verification activities are required by safety standards ecss40C . Further, although embedded software for CPS can be developed using dedicated visual languages Buffoni2021 , our experience in the space context shows that most software is directly implemented using C and C++, which suggests that approaches supporting validation and verification shall target those languages. Mutation analysis is an effective method to evaluate the quality of a test suite. It involves measuring the mutation score, which is the proportion of programs with artificially injected faults (mutants) detected by the test suite papadakis2019mutation . There is a strong association between a high mutation score and a high fault-revealing capability for test suites papadakis2018mutation ; Chekam:17 . Additionally, recent studies have demonstrated that mutation analysis can be applied cost-effectively to large CPS software Oscar:TSE . In practice, mutation analysis warrants the selection of inputs for mutation testing, as test cases should ideally detect all or at least a significant proportion of the generated mutants. A mutant detected by a test suite is considered killed. However, due to the typically high number of mutants generated in large CPS projects, it is challenging for engineers to perform mutation testing manually. Unfortunately, we lack automated test data generation techniques ( automated mutation testing techniques) suitable for the mutation testing of CPS. Indeed, most existing techniques do not target C and C++, which are prevalent in CPS. Further, the state-of-the-art solution for automated mutation testing of C software, SEMu Chekam2021 , is based on the KLEE symbolic execution engine KLEE . While effective for command line utilities, it inherits the limitations of symbolic execution. Specifically, it requires environmental modeling (e.g., network communication) and cannot be applied to programs needing complex analyses for input generation (e.g., programs with floating point instructions). Additionally, it generates test inputs for command line utilities, which are rarely used in CPS, and does not produce unit test cases or target other CPS interfaces. Search-based techniques developed for other programming languages (e.g., Java fraser2011mutation ) are impractical for C and C++ software due to the difficulty of instrumenting the software to compute dedicated fitness functions (e.g., branch distance). For instance, computing branch distance at runtime necessitates modifying all conditional statements in the software under test (SUT), requiring the source code to be processed with static analysis tools that load all dependencies. Configuring these tools to process the multiple source files in large systems is often impractical unless the tool is well integrated with the compiler used for the SUT. Moreover, CPS source files often rely on architecture-specific C constructs (e.g., for the RTEMS compiler RTEMS ) that are not successfully parsed by static analysis frameworks Oscar:TSE . Different from search-based testing techniques, grey-box fuzz testing techniques manes2019art (hereafter, fuzzers ) can generate test data without relying on complex code instrumentation and thus can be easily applied to C and C++ software. However, fuzzers target console software, while CPS software is usually tested either with system-level test scripts interacting with a hardware emulator or through unit and integration test cases implemented with the same language as the SUT. In this paper, we focus on the automated generation of unit test cases because fuzzing large systems is an open research problem KimXSO22 . Approaches for fuzzing function calls at unit or API level exist FuzzGen ; FUDGE ; APICRAFT ; Jeong2023 ; Hopper , although in the case of large CPS software, their applicability might be limited by static analysis’ scalability issues and difficulties in processing embedded libraries. Further, approaches to generate fuzzing drivers for mutation testing do not exist. We propose MutatiOn TestIng with Fuzzing ( MOTIF ), an approach that automatically generates fuzz drivers, leverage the data produced by fuzzers, and exercise a mutated function and the corresponding original function, looking for diverging outputs to detect mutants. MOTIF automatically generates seed files for fuzzers and integrates strategies to process mutant-killing inputs to eliminate false positives due to nondeterminism. Instead of designing a dedicated fuzzing algorithm, MOTIF leverages state-of-the-art (SOTA) fuzzers because we believe that the fuzzers’ bucket-based coverage strategy can effectively select test inputs that eliminate mutants. This strategy not only selects inputs that lead to different software states but also tracks and is guided by the behavioral differences between the original and mutated functions. Although MOTIF can target any C/C++ program, we designed it taking into account characteristics that are prominent in embedded software for CPS. Specifically, CPS software is not executed from the textual console but directly within the operating system, largely relies on floating point instructions, and leverages specific OS and libraries that complicates its analysis or symbolic interpretation. For these reasons, we present MOTIF as a tool for C/C++ embedded software for CPS. We introduced MOTIF in a paper presented at the 2023 International Conference on Automated Software Engineering MOTIF where we reported on its effectiveness on three case study subjects in the space domain developed using the C language and demonstrated that it outperforms mutation testing based on symbolic execution. In this paper, we extend our previous work by: • Performing a large empirical study to investigate (RQ1) the best fuzzing configurations (compiler, sanitizers, and coverage metric) for mutation testing. Note that the identification of the best fuzzing configuration for different application contexts remains an open problem EURECOM7801 . Our experiments show that the best results are obtained when relying (1) on the Clang compiler, which leads to the quickest fuzz drivers, (2) the address sanitizer ( ASAN ), which prevents false positives due to violation of function preconditions, and (3) the LAF-Intel LAF:Blog optimization for coverage, which enables computing a better fitness score for mathematical functions, which

Read the complete text