Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection
Max Hort, Linas Vidziunas, Leon Moonen · 2025
Read Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection by Max Hort, Linas Vidziunas, Leon Moonen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23448.txt.
Introduction
Max Hort, Linas Vidziunas, and Leon Moonen test whether meaning-preserving code rewrites can sharpen bug-finding tools at test time. The idea mirrors metamorphic testing: a tool should give the same verdict on code that means the same. Their 2025 study first gathered published rewrite tools, finding ninety-four rewrites across twenty-eight papers. They built thirty-nine of them, then hand-checked each one.
The check was sobering: twenty-three of the thirty-nine rewrites changed code meaning, so only sixteen were safe. With those, three ensemble voting schemes were tried on a flaw data set with two language models. Accuracy did not rise. The authors warn that shared rewrites are hard to reuse safely.
Catalog record
| CREATOR | Max Hort, Linas Vidziunas, Leon Moonen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection Max Hort, Linas Vidziunas, Leon Moonen Abstract Recent advances in defect detection use language models. Existing works enhanced the training data to improve the models’ robustness when applied to semantically identical code (i.e., predictions should be the same). However, the use of semantically identical code has not been considered for improving the tools during their application - a concept closely related to metamorphic testing. The goal of our study is to determine whether we can use semantic-preserving transformations, analogue to mutation operators, to improve the performance of defect detection tools in the testing stage. We first collect existing publications which implemented semantic-preserving transformations and share their implementation, such that we can reuse them. We empirically study the effectiveness of three different ensemble strategies for enhancing defect detection tools. We apply the collected transformations on the Devign dataset, considering vulnerabilities as a type of defect, and two fine-tuned large language models for defect detection (VulBERTa, PLBART). We found 28 publications with 94 different transformation. We choose to implement 39 transformations from four of the publications, but a manual check revealed that 23 out 39 transformations change code semantics . Using the 16 remaining, correct transformations and three ensemble strategies, we were not able to increase the accuracy of the defect detection models. Our results show that reusing shared semantic-preserving transformation is difficult, sometimes even causing wrongful changes to the semantics. I Introduction Among others, defect detection models should be accurate ( good at detecting defective code snippets ) and robust ( making the same prediction for functions that are doing the same thing ). Existing research measured how robust defect detection models are [ 1 ] and improved robustness via the training process (e.g., adversarial training) by incorporating code snippets that look different but have the same functionality [ 1 , 2 ] . One approach to obtain such snippets is the use semantic-preserving transformations (e.g., changing a for loop to a while loop), which do not change the code snippets’ functionality. An unexplored aspect is whether “unrobust” models can be improved by using the diverse predictions made on code transformed with semantic-preserving transformations. This intuition is inspired by metamorphic testing, which instead of ground truth datasets, has access to inputs which should generate the same output [ 3 , 4 ] . Models are then evaluated by their ability to make the same predictions for such input pairs. Rather than comparing the predictions made on multiple inputs, as done in metamorphic testing, we use them to improve the performance of LLMs for defect detection. As an example, imagine you have written code which a defect detection model predicts as “correct”. You now go ahead and change a variable name. All of a sudden the prediction changes to defective. Surprised, you repeat the same procedure another 5 times but the prediction remains as defective. This raises the question whether to trust the first prediction, as it was performed on the original code snippet or the majority of tests, which were labeled as defective. Following this intuition, we study the following: What is the impact of semantic-preserving transformations on defect detection tool performance? To answer this question, we focus on Large Language Models (LLMs) as defect detection tools. First, we review existing literature that applied semantic-preserving transformations. Then we check whether they share resources to find transformations we can replicate and apply for our experiments. Using LLMs as defect detection models and existing approaches for generating semantic-preserving transformations, we focus on combining the two parts and investigate the ability of semantic-preserving transformations to enhance and improve the defect detection models. For this purpose, we use ensemble techniques, which make predictions under consideration of the original code snippet and mutated variants with semantic-preserving transformations. Figure 1 outlines the current process of using LLMs for defect detection and our novel approach, using semantic-preserving transformations for mutating source code, and ensemble learning. We apply the ensembles to the Devign dataset [ 5 ] , a dataset for detecting vulnerability defects, which categorises functions as vulnerable and non-vulnerable. Fig. 1: Overview of the proposed approach for applying semantic-preserving transformation for testing defect detection models. We make the following contributions: • We collect 28 publications that applied semantic-preserving transformations. • We implement 39 semantic-preserving transformations shared by four publications. For each transformation, we check up to 20 transformed functions manually to verify their correctness. • We evaluate the usability of semantic-preserving transformations for improving defect detection tools in an empirical study with three ensemble techniques and two LLMs using the transformations. • We share our results and all transformations applied to the Devign dataset to enable a replication and verification of our results. 1 1 1 https://figshare.com/s/f9b93d9d549f316b1e5d . Main findings: We have found 28 publications with 94 semantic-preserving transformations. When implementing the transformations of four of these repositories, we encountered difficulties. Critically, 23 out of 39 transformations did not pass our manual validation, as they changed semantics . We are left with 16 out of 39 transformations for our empirical study. While our investigated ensemble techniques were not able to improve the accuracy, we provide insights on implementation difficulties (Section V-C ). II Related Work II-A Data Augmentation for Software Engineering Data augmentation is used to increase the amount of data available for training or testing of machine learning models. Here we outline approaches for software engineering tasks. Sampling: The first set of approaches are concerned with the sampling of existing instances in the training dataset. When treating sampling as a data augmentation approach, we focus on the oversampling of data (i.e., the addition of duplicates) rather than downsampling (i.e., removal of samples) [ 6 , 7 ] . Sampling is in particular useful when dealing with imbalanced datasets, which frequently is the case for vulnerability detection tasks. For instance, datasets might only have 10% of functions labeled as vulnerable [ 8 ] . When collecting programs from online sources, such as Github, the proportion of correct samples is much larger than defective programs. Oversampling can balance the two classes [ 9 ] . In addition to random sampling, more complex methods have been applied, such as fuzzy-based oversampling [ 10 ] or Synthetic Minority Over-Sampling (SMOTE) [ 11 , 12 ] . SMOTE selects a sample from the under-represented group and one of its closest neighbors (from the same group). The new sample is created by interpolating between the two selected data samples [ 13 ] . For such an interpolation to work, one needs to represent programs and code snippets numerically. This has been done with vector embeddings [ 12 , 11 ] or based on source code metrics [ 14 , 15 , 16 ] Regular oversampling is not able to generate new programs. SMOTE can generate new data but represents programs as vector embeddings rather than source code. Neural Approaches: Neural approaches use LLMs to learn how to modify existing code snippets and create new snippets from scratch. For example, Richter and Wehrheim [ 17 ] used LLMs to learn how to mutate tokens in a code snippet to generate more natural bugs. Another approach has been followed by Zirak