WITNESS: A lightweight and practical approach to fine-grained predictive mutation testing DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Software testing and debugging
Zeyu Lu, Peng Zhang, Chun Yong Chong, Shan Gao, Yibiao Yang, Yanhui Li, Lin Chen, Yuming Zhou · 2025
Read WITNESS: A lightweight and practical approach to fine-grained predictive mutation testing DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Software testing and debugging by Zeyu Lu, Peng Zhang, Chun Yong Chong, Shan Gao, Yibiao Yang, Yanhui Li, Lin Chen, Yuming Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11999.txt.
Introduction
Zeyu Lu and seven co-authors make predictive mutation testing cheaper and broader with WITNESS. Mutation testing grades a test suite by seeding bugs. It checks if the tests catch them. Predictive versions guess mutant outcomes to skip costly runs. Deep learning predictors cost so much they defeat that purpose.
They also handle only mutants inside methods. Real tools generate mutants outside methods too. A lightweight approach replaces the heavy models. It covers outside-method mutants as well, restoring real-world use. Fine-grained prediction at each mutant location keeps the feedback developers use. Test quality tooling helps only if teams can afford to run it.
Catalog record
| CREATOR | Zeyu Lu, Peng Zhang, Chun Yong Chong, Shan Gao, Yibiao Yang, Yanhui Li, Lin Chen, Yuming Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 125 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11999
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WITNESS: A lightweight and practical approach to fine-grained predictive mutation testing DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Software testing and debugging Zeyu Lu, Peng Zhang, Chun Yong Chong, Shan Gao, Yibiao Yang, Yanhui Li, Lin Chen, Yuming Zhou Abstract . Background: Predictive mutation testing mitigates the high computational cost of traditional mutation testing by employing prediction models to approximate mutation testing results. In its fine-grained form, predictive mutation testing predicts whether individual test cases will kill specific mutants. These predictions enable the construction of kill matrices, which is essential for downstream tasks such as test case prioritization. Problem: Existing fine-grained predictive mutation testing studies predominantly rely on deep learning, which faces two critical limitations in practice: (1) Exorbitant computational costs. The deep learning models adopted in these studies demand significant computational resources for training and inference acceleration. This introduces high costs and undermines the cost-reduction goal of predictive mutation testing. (2) Constrained applicability. Although modern mutation testing tools generate mutants both inside and outside methods, current fine-grained predictive mutation testing approaches handle only inside-method mutants. As a result, they cannot predict outside-method mutants, limiting their applicability in real-world scenarios. Objective: We aim to develop a more cost-effective and practically applicable fine-grained predictive mutation testing approach that reduces computational costs while supporting the prediction of outside-method mutants, thereby enhancing its real-world applicability. Method: We propose WITNESS, a new fine-grained predictive mutation testing approach. WITNESS adopts a twofold design: (1) With collected features from both inside-method and outside-method mutants, WITNESS is suitable for all generated mutants. (2) Instead of using computationally expensive deep learning, WITNESS employs lightweight classical machine learning models for training and prediction. This makes it more cost-effective and enabling straightforward explanations of the decision-making processes behind the adopted models. Result: Evaluations on Defects4J projects show that WITNESS consistently achieves state-of-the-art predictive performance across different scenarios. Additionally, WITNESS significantly enhances the efficiency of kill matrix prediction. Post-hoc analysis reveals that features incorporating information from before and after the mutation are the most important among those used in WITNESS. Test case prioritization based on the predicted kill matrix shows that WITNESS delivers results much closer to those obtained by using the actual kill matrix, outperforming baseline approaches. Conclusion: Compared to deep learning-based fine-grained predictive mutation testing, WITNESS is simpler yet delivers higher effectiveness, greater efficiency, and broader applicability. It advances fine-grained predictive mutation testing and sets a new baseline for future research. 1. Introduction Mutation testing, which was first proposed by DeMillo ( 9 ) , is widely used to evaluate the effectiveness of test suites. Mutation testing employs mutation operators to modify the original program under test, creating many mutants. These modifications are small, making the mutants identical to the original program except for the changed statements. The test suite is executed against each mutant to detect differences in output compared to the original program. If a test case causes a mutant to produce different outputs, the mutant is considered “killed.” Conversely, if no test case can produce different outputs, the mutant “survives.” The effectiveness of the test suite is measured by the mutation score, which is the ratio of killed mutants to the total number of mutants. A higher mutation score indicates a more effective test suite, while a lower score suggests weaker defect-detection capability. Although mutation testing is more effective than code coverage in evaluating test suite effectiveness ( 63 ; 41 ; 13 ; 4 ; 25 ) , it is less commonly adopted in practice ( 51 ) . The primary reason for this is the high cost associated with mutation testing. Without optimization, the test suite must be executed on each mutant to calculate the mutation score. Even a small program could generate many mutants ( 69 ) , and executing the test suite against all generated mutants requires substantial time, especially for larger projects. To reduce the cost of mutation testing, many approaches have been proposed. Mutant reduction ( 17 ; 68 ) is one example. By decreasing the number of executed mutants, the cost of mutation testing can be reduced. However, even with mutant reduction, the execution of the test suite against some generated mutants is still required, keeping the costs relatively high ( 37 ) . Predictive mutation testing uses machine learning models to predict whether a mutant will be killed, offering a more cost-effective alternative to traditional mutation testing. This approach was first introduced by Zhang et al. ( 67 ) , who used a Random Forest model to predict whether a mutant will be killed by the entire test suite based on specific features. However, predicting mutant killing outcomes does not fully capture the relationship between individual mutants and test cases. To tackle this problem, three recent studies proposed Seshat ( 32 ) , MutationBERT ( 23 ) , and SODA ( 72 ) , to predict whether a specific test case can kill a given mutant, generating a predicted kill matrix for all mutant-test case pairs. This finer-grained approach allows for more detailed downstream tasks, such as mutation-based test case prioritization using the Additional approach ( 54 ) . Despite these advancements, Seshat, MutationBERT, and SODA all have notable limitations. 1 High computational cost. These approaches use deep learning models that entail high computational demands for efficiency. Such demands introduce higher costs, reduced accessibility, and more complex setup and maintenance, contradicting the cost-reduction goal of predictive mutation testing. For example, the total computational expense we incurred for conducting experiments across all three studies ( 32 ; 23 ; 72 ) in this paper exceeded $3,000. Furthermore, even with the aid of computational resources, the prediction process remains time-consuming; for instance, we found that MutationBERT’s prediction time exceeded the time required for actual mutation testing in several experiments on our machine. 2 Constrained applicability. These approaches are applicable only to mutants where the mutation occurs within source methods. As they rely on the names and parameters of source methods, MutationBERT and SODA also collect contextual statements surrounding a mutated statement within the same source method. Outside-method statements—whose types typically include field declarations, static initializer blocks, instance initializer blocks, and definitions of inner classes, enums, or interfaces—are essential for defining the structure, state, and initialization behavior of Java classes ( 2 ) . Existing mutation testing tools ( 26 ; 7 ; 52 ; 35 ) for Java do not restrict their applicability to inside-method statements alone. For example, for mutants generated by Major ( 26 ) , a mutation testing tool commonly used by Seshat, MutationBERT, and SODA, the mutant–test pairs with mutations occurring outside source methods can account for up to 24.35% of all mutant–test pairs in the JacksonCore_5 project used in our study. This significantly limits the utility of these approaches. To address the limitations of the three approaches that predict the kill matrix, we propose WITNESS (light- W e I ght T est effective NES s mea S urement), a lightweight approach that evaluates test suite effectiveness by predicting the kill matrix. WITNESS adopts a dual-track design: (1) With collected features tailored to both inside-method and outside-method mutants, WITNESS is suitable for all generated mutants, thereby facilitating its practical use. (2) WITNESS employs lightweight classical machine