Boosting LLMs for Mutation Generation DOI: XXXXXXX.XXXXXXX Conference: The ACM International Conference on the Foundations of Software Engineering (FSE); July 5–9, 2026; Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging
Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang · 2026
Read Boosting LLMs for Mutation Generation DOI: XXXXXXX.XXXXXXX Conference: The ACM International Conference on the Foundations of Software Engineering (FSE); July 5–9, 2026; Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging by Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24560.txt.
Introduction
Wang and co-authors present SMART, a system that boosts language models at generating software mutations for testing. Earlier tools used fixed few-shot examples or none, yielding generic or uncompilable mutants. SMART combines retrieval over a vectorised dataset of real-world bugs, focused code chunking, and fine-tuning on mutations paired with real bugs. A study on 1,991 real Java bugs from Defects4J and ConDefects compares it with two rival tools. SMART improves mutant validity and efficiency, and small 7B models match or beat GPT-4o, the authors report.
Catalog record
| CREATOR | Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69520 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24560
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Boosting LLMs for Mutation Generation DOI: XXXXXXX.XXXXXXX Conference: The ACM International Conference on the Foundations of Software Engineering (FSE); July 5–9, 2026; Montreal, Canada ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang Abstract . LLM-based mutation testing is a promising testing technology, but existing approaches typically rely on a fixed set of mutations as few-shot examples or none at all. This can result in generic low-quality mutations, missed context-specific mutation patterns, substantial numbers of redundant and uncompilable mutants, and limited semantic similarity to real bugs. To overcome these limitations, we introduce SMART (Semantic Mutation with Adaptive Retrieval and Tuning). SMART integrates retrieval-augmented generation (RAG) on a vectorized dataset of real-world bugs, focused code chunking, and supervised fine-tuning using mutations coupled with real-world bugs. We conducted an extensive empirical study of SMART using 1,991 real-world Java bugs from the Defects4J and ConDefects datasets, comparing SMART to the state-of-the-art LLM-based approaches, LLMut and LLMorpheus. The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency (even enabling small-scale 7B-scale models to match or even surpass large models like GPT-4o). We also demonstrate that SMART significantly improves downstream software engineering applications, including test case prioritization and fault localization. More specifically, SMART improves validity (weighted average generation rate) from 42.89% to 65.6%. It raises the non-duplicate rate from 87.38% to 95.62%, and the compilable rate from 88.85% to 90.21%. In terms of effectiveness, it achieves a real bug detection rate of 92.61% (vs. 57.86% for LLMut) and improves the average Ochiai coefficient from 25.61% to 38.44%. For fault localization, SMART ranks 64 more bugs as Top-1 under MUSE and 57 more under Metallaxis. 1. Introduction Mutation testing ( 19 ; 13 ) is widely regarded as one of the most fundamental dynamic techniques in software testing and program analysis. By systematically introducing artificial faults, known as mutations , into the program under test, mutation testing enables researchers and practitioners to assess the adequacy of test suites ( 25 ) , prioritize effective test suites ( 16 ; 33 ; 44 ; 6 ) , or localize buggy statements/methods ( 35 ; 39 ; 52 ) . In recent years, mutation testing has gained widespread adoption in the industry. Google and Meta have reported their successful experience in deploying mutation testing ( 21 ; 40 ; 3 ) . The effectiveness of mutation testing, however, fundamentally depends on the quality of the generated mutations. Generating effective and diverse mutants is thus critical not only for evaluating test suites, but also for supporting downstream applications such as test case prioritization and fault localization. Recently, the rapid advancement of Large Language Models (LLMs) has opened new opportunities for mutation testing ( 54 ; 7 ) . Due to their strong code understanding and generation capabilities, LLMs have been explored to automatically generate more realistic and semantically meaningful mutants. Both academic research ( 12 ; 23 ; 48 ; 50 ) and industrial practice ( 21 ) have begun to investigate LLM-based mutation generation as a promising alternative to traditional approaches. Despite encouraging progress, existing LLM-based approaches still suffer from several major limitations. First, they rely on either a fixed set of few-shot examples as in-context examples or no examples at all, making it difficult for models to adapt to diverse program contexts. For example, BugFarm ( 23 ) and LLMorpheus ( 48 ) rely solely on prompts, whereas LLMut ( 50 ) incorporates a fixed set of real-world bugs as few-shot examples to instruct LLMs. Second, they typically restrict the code context to the surrounding focal method, which blinds the model to crucial program-wide dependencies. All existing LLM-based mutation generation approaches directly adopt the whole focal method as context. Due to these limitations, the generated mutations still contain a substantial proportion of redundant and uncompilable cases, with limited semantic similarity to real bugs ( 50 ) . To address these limitations, we present SMART ( S emantic M utation with A daptive R etrieval and T uning), a novel LLM-based mutation generation approach that integrates retrieval-augmented generation (RAG) ( 32 ) , code chunking, and fine-tuning ( 5 ) . Our approach is designed to enhance both the relevance and effectiveness of generated mutations. First, to select context-aware few-shot examples, we build a vectorized dataset of real-world bugs and employ RAG to retrieve the most relevant “bug–fix” pairs for the target method. We collect 130,000 real-world single-hunk bugs with corresponding patches, and embed the patched versions as the keys for retrieval. This ensures that the LLM benefits from in-context examples that are semantically aligned with the mutation task at hand. Second, rather than mutating the entire method in a monolithic way, we partition the focal method into function-coherent code chunks and generate mutations at the chunk level, which improves focus and diversity. Finally, to adapt LLMs more closely to the task of producing semantically effective mutations, we apply supervised fine-tuning (SFT) using effective mutations that are coupled with real-world bugs. The training set comprises 13,760 mutations that are coupled with real bugs from the prior study ( 50 ) . To prevent data leakage, all the training mutations are from projects beyond our evaluation. By combining retrieval, structural code decomposition, and model adaptation, SMART aims to advance the state of LLM-based mutation generation significantly. To assess the quality of the mutations generated by SMART, we conducted an extensive empirical study on 1,991 real-world Java bugs, including 701 from Defects4J ( 27 ) and 1,290 from ConDefects ( 58 ) . We evaluated SMART using five open-source LLMs (DeepSeek-Coder-6.7B, Llama-3.1-8B, Qwen-2.5-7B, Qwen-2.5-14B, and Qwen-2.5-32B), and additionally tested GPT-4o with RAG and code chunking enabled. For comparison, we included two state-of-the-art LLM-based mutation generation approaches: LLMorpheus ( 48 ) and LLMut ( 50 ) . Our evaluation covers a wide range of dimensions. We first examine the effectiveness of generated mutations, i.e., how closely they couple with real-world bugs. We then assess their validity, since LLMs inevitably produce redundant or incompilable mutants. Finally, we evaluate their impact on downstream applications, including mutation-based test case prioritization and fault localization. The results show that SMART substantially improves both the validity and effectiveness of LLM-generated mutations. For validity, SMART raises the weighted average generation rate from 42.89% (LLMut) to 65.6% ( \uparrow 52.95\% ; +22.71 PPs), increases the non-duplicate rate from 87.38% (LLMut) and 85.87% (LLMorpheus) to 95.62% ( \uparrow 9.42\% –11.36%), and improves the compilable rate from 88.85% (LLMut) and 78.43% (LLMorpheus) to 90.21% ( \uparrow 1.53\% –15.0%). For effectiveness, SMART achieves a weighted average real bug detection rate of 92.61%, compared to 57.86% for LLMut and 31.99% for LLMorpheus, and raises the average Ochiai coefficient to 38.44% ( \uparrow 50.1\% and 277.6% over baselines). On downstream applications, SMART enhances mutation-based fault localization by ranking more bugs in Top- k and reducing MAR/MFR (e.g., 143 Top-1 bugs by MUSE vs. 106 for LLMut and 11 for LLMorpheus). Notably, our approach enables 7B-scale models to achieve performance competitive with GPT-4o, making it suitable for scenarios with limited computational resources. In the discussion section, we further analyze the computational cost and robustness