An Empirical Study of the Realism of Mutants in Deep Learning

Zaheed Ahmed, Philip Makedonski, Jens Grabowski · 2025

Read An Empirical Study of the Realism of Mutants in Deep Learning by Zaheed Ahmed, Philip Makedonski, Jens Grabowski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16741.txt.

Introduction

Zaheed Ahmed, Philip Makedonski, and Jens Grabowski run the first head-to-head test of whether fake faults in deep learning act like real faults. Testers use mutants, small edits to a model or its training, as stand-ins for real bugs, but that swap was unverified for deep learning.

The team scored realism with two measures, coupling strength and behavior match to real faults, across four public bug sets: CleanML, DeepFD, DeepLocalize, and defect4ML. Mutants made before training beat those made after training on both scores, across single operators too. The catch is cost: pre-training mutants take far more compute, so better post-training operators are still needed.

Catalog record

CREATORZaheed Ahmed, Philip Makedonski, Jens Grabowski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study of the Realism of Mutants in Deep Learning Zaheed Ahmed, Philip Makedonski, Jens Grabowski Abstract Mutation analysis is a well-established technique for assessing test quality in the traditional software development paradigm by injecting artificial faults into programs. Its application to deep learning (DL) has expanded beyond classical testing to support tasks such as fault localization, repair, data generation, and model robustness evaluation. The core assumption is that mutants behave similarly to real faults, an assumption well established in traditional software systems but largely unverified for DL. This study presents the first empirical comparison of pre-training and post-training mutation approaches in DL with respect to realism. We introduce a statistical framework to quantify their coupling strength and behavioral similarity to real faults using publicly available bugs datasets: CleanML , DeepFD , DeepLocalize , and defect4ML . Mutants are generated using state-of-the-art tools representing both approaches. Results show that pre-training mutants exhibit consistently stronger coupling and higher behavioral similarity to real faults than post-training mutants, indicating greater realism. However, the substantial computational cost of pre-training mutation underscores the need for more effective post-training operators that match or exceed the realism demonstrated by pre-training mutants. ∎ 1 Introduction Deep learning (DL) systems are increasingly deployed in domains where model reliability is critical, which has driven extensive research on testing and debugging techniques for DL. Many of these techniques rely on mutation analysis to create synthetic faulty versions of models, training scripts, or data for evaluation purposes. These mutants often serve as substitutes for real faults. However, whether these mutants faithfully resemble real DL faults remains unclear, which raises concerns about the validity of mutation-based evaluations. At the same time, real faults are difficult to obtain at scale: 27 report that only 3.48% of GitHub issues and 2.93% of Stack Overflow posts involving DL faults were reproducible, and 17 were able to reproduce only 52% of faults in curated datasets. The limited availability and reproducibility of real faults in DL research have made mutation testing a common substitute. In the traditional software development paradigm, mutation testing ( 29 ) tools provide an alternate means to generate faulty program versions by injecting small syntactical changes. These modified versions, called mutants, are often used as proxies for real faults. The primary motivation for using mutants as substitutes for real faults lies in their perceived realism. The theory of the realism of mutants in mutation testing is grounded in two foundational hypotheses: the competent programmer hypothesis, and the coupling effect ( 8 ) . On one hand, the competent programmer hypothesis assumes that programmers usually write nearly correct code. This means that small syntactic changes introduced by mutation operators can reflect developers’ mistakes, as explored in our recent study ( 4 ) . On the other hand, 8 stated the coupling effect as: “Test data that distinguishes all programs differing from a correct one by only simple errors is so sensitive that it also implicitly distinguishes more complex errors” . This implies that mutants are useful when they are coupled to real faults, as test inputs that kill the mutant may also detect the corresponding real fault. Empirical evidence from traditional software development strongly supports this assumption, with studies reporting high coupling rates between mutants and real bugs across diverse datasets and environments ( 20 ; 30 ; 22 ) . Recently, mutation analysis has been adapted for the machine learning paradigm ( 25 ; 33 ; 12 ; 14 ; 21 ) . Beyond test assessment, mutation analysis in DL has been employed across a broad range of tasks including automated program repair ( 39 ; 34 ) , input data prioritization for labeling ( 36 ) , robustness evaluation of models ( 12 ; 24 ; 26 ) , evaluation of data cleaning techniques ( 23 ; 2 ) , generation and detection of adversarial examples ( 35 ; 31 ) , test data generation ( 32 ; 9 ) , oracle generation for autonomous vehicles ( 18 ) , fault localization in deep neural networks ( 6 ; 10 ) , and supporting metamorphic testing strategies in machine learning ( 40 ) . This wide range of use cases underscores the importance of ensuring that DL mutants behave realistically. The realism of mutants has far-reaching implications for the broader DL engineering ecosystem. If mutants deviate substantially from the behavior of real faults, these downstream techniques risk drawing misleading conclusions. Establishing empirical evidence on the realism of DL mutants is therefore essential to ensure that mutation-based evaluation and analysis methods remain credible across diverse applications. Unlike traditional software systems, mutant detection in DL is often probabilistic due to inherent nondeterminism. Therefore, the fundamental hypotheses of classical mutation testing do not directly translate to this new data-driven development paradigm ( 28 ) . We need to carefully reconsider their validity. In DL, mutants can be generated either via pre-training or post-training ( 25 ; 3 ) ; however, which approach yields mutants more closely coupled with real faults remains an open question. To date, no prior work has systematically investigated the extent to which DL mutants represent real faults. In this study, we aim to quantify and compare the degree of realism in pre-training and post-training mutants of DL systems. We propose a statistical method to measure the coupling strength and behavioral similarity between the mutants and the real faults. To account for nondeterminism, each mutant model is trained multiple times, and evaluated on every test input to compute per-input killing probabilities. Based on these probabilities, we quantify the realism in terms of the coupling strength and behavioral similarity of each mutant with its corresponding real fault. This paper makes the following key contributions: • We curate a dataset of reproducible real faults in DL systems, selected from multiple public datasets. We systematically filter out crash-inducing bugs and retain only those faults that (i) exhibit observable behavioral changes, (ii) include test or validation data, and (iii) can be executed in a unified, controlled environment. All bugs were syntactically adapted where necessary to ensure compatibility and reproducibility. • We propose a statistical methodology to quantify the relationship between DL mutants (pre-training and post-training) and real faults, enabling a rigorous assessment of mutant representativeness. Our approach introduces two complementary metrics to evaluate the coupling strength and behavioral similarity in a nondeterministic DL context. The remainder of this paper is organized as follows. Section 2 provides background on mutation testing for DL. Section 3 presents our proposed approach to the problem. Section 4 describes the evaluation methodology used in this study. Section 5 reports the experimental results, followed by a discussion in Section 6 . Finally, Section 7 concludes the paper and outlines directions for future work. 2 Background This section first outlines how mutation testing has been adapted to DL, emphasizing the main mutation targets and approaches. It then summarizes prior studies evaluating the effectiveness and potential realism of DL mutants. 2.1 Mutation Testing for Deep Learning Mutation analysis has emerged as a crucial methodology for assessing the quality and robustness of DL models ( 25 ; 33 ; 12 ; 14 ; 21 ) . Unlike traditional software development, where logic is manually coded,

Read the complete text