PITMuS : A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction

Tasfia Tasnim, Soneya Binta Hossain · 2026

Read PITMuS : A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction by Tasfia Tasnim, Soneya Binta Hossain (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21930.txt.

Introduction

Tasfia Tasnim and Soneya Binta Hossain present PITMuS, a tool that turns bytecode-level mutation reports from the Java tester PIT into source-level mutant pairs. PIT reports mutation metadata rather than the source edits, so its mutants are hard to inspect, replay, and reuse. PITMuS reconstructs the original and mutant source methods for each report. From 69,229 reported mutations across 1,913 files, it produces 69,198 source-level pairs, covering 99.96 percent of the records for testing, repair, and learning-based software work.

Catalog record

CREATORTasfia Tasnim, Soneya Binta Hossain
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16726
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PITMuS : A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction Tasfia Tasnim, Soneya Binta Hossain Abstract . LLM-based software engineering increasingly depends on executable, context-rich bug artifacts: paired correct and buggy code, methods under test (MUTs), documentation, and metadata. These artifacts support the training and evaluation of automated bug localization and repair techniques, testing and test oracle generation methods, and documentation-driven automation. Although curated benchmarks (e.g., Defects4J) remain valuable, they are static and increasingly vulnerable to contamination as code models are trained on large public corpora. A complementary strategy is to generate fresh, cutoff-aware datasets by selecting real system versions and injecting controlled bugs at the source level. Mutation testing is a natural basis for this strategy: it applies predefined mutation operators to programs and records whether the existing test suite detects each injected change. PIT is a state-of-the-practice mutation testing tool for Java that performs mutation at the bytecode level. This design makes mutation testing fast and practical, but PIT reports mutants primarily through XML, making them difficult to inspect, replay, or reuse as structured source-level dataset records. To address this gap, we present PITMuS , which combines PIT XML metadata with debug information from compiled Java class files to localize and reconstruct the source edit corresponding to each mutant. PITMuS then automatically produces structured datasets containing source-level buggy and fixed code pairs, documentation context, and metadata for downstream training and evaluation. Although we evaluate PITMuS on eight open-source Java systems, it can be applied to any Java system where PIT can be integrated. Screencast: https://youtu.be/zgHkXnsgciw Artifact: https://github.com/assert-lab/PITMuS 1. Introduction Automated and LLM-based software engineering tasks such as bug localization, program repair, test generation, test oracle generation and assessment, and documentation-driven automation require paired examples of correct and buggy code, along with methods under test (MUTs), documentation, localization information, and metadata describing the fault. Such artifacts make it possible to train and evaluate techniques on concrete source-level changes within their surrounding program context. Curated benchmarks such as Defects4J ( 8 ) have been central to software engineering research, but they are fixed by design. They cover a limited set of systems and bugs, and they typically do not evolve as new system versions become available or as the field shifts toward new paradigms such as LLM-based software engineering. This limitation is especially pressing in the LLM era because code models are trained on large public corpora, making widely used benchmarks increasingly vulnerable to contamination ( 7 ; 12 ; 13 ) . A complementary strategy is to generate fresh, cutoff-aware datasets by selecting real system versions and injecting controlled bugs at the source level. Mutation testing is a natural basis for this strategy: it applies predefined mutation operators to programs and records whether the existing test suite detects each injected change ( 2 ) . In the Java ecosystem, PIT ( 10 ) is a state-of-the-practice mutation testing tool because it is fast, integrates cleanly with modern build workflows, and has been used in numerous software engineering studies ( 9 ; 3 ; 5 ; 4 ) . PIT achieves this practicality by performing mutation at the bytecode level and reporting each mutant through metadata such as the mutator, source file, line number, and mutation description. Figure 1. Example of a source line, a mutation applied to it, and the resulting mutated source line. This design is effective for mutation-score computation, but it limits the usefulness of PIT mutants as source-level artifacts. A PIT report tells users that a mutant exists, where it occurred, and what mutation operator was applied, but it does not directly provide the mutated source code. Consequently, users cannot easily inspect the mutant in an editor, replay it as a concrete source file, compile it, or reuse it as a structured dataset record containing the original code, mutated code, documentation context, and mutation metadata. Recovering such artifacts is, to the best of our knowledge, neither readily supported by existing tools nor always straightforward. A line number and mutation description may be ambiguous when a source line contains multiple candidate operators. For example, a line containing both index + 1 and length + index has two possible targets for a mutation described as “Replaced integer addition with subtraction.” PIT distinguishes such mutants using bytecode-level information, but that information must be reconciled with the original source code to reconstruct the exact edit. To address this gap, we present PITMuS , a tool for automated dataset generation via source-level mutant reconstruction. PITMuS combines PIT XML metadata, debug information from compiled Java class files, and javalang ( 11 ) parsing of the original source code to localize and reconstruct the source edit corresponding to each mutant. PITMuS then produces reusable source-level artifacts and structured datasets containing buggy and fixed code pairs, documentation context, and mutation metadata. This work makes the following contributions: • We introduce PITMuS , which reconstructs source-level mutants from standard PIT reports without replacing the underlying mutation engine. • We provide two utilities: source-level mutant injection and replay, and method-level dataset construction with original methods, mutated methods, Javadoc when available, and mutation metadata. • We evaluate PITMuS on eight open-source Java systems, where it processes 69,229 PIT mutations and reconstructs 69,198 aligned mutant-method pairs across 1,913 source files. Although evaluated on these systems, PITMuS can be applied to any Java system where PIT can be integrated. 2. PITMuS Workflow PITMuS reconstructs source-level mutants from three inputs: a PIT XML report, the system’s compiled .class files, and the original Java source files. The XML report identifies each reported mutant, while the class files and source files provide the information needed to map bytecode-level mutations back to concrete source edits. Figure 2 summarizes the high-level workflow. Figure 2. Overview of PITMuS workflow. PITMuS can be invoked through two commands: python gen_dataset.py <system-path> for dataset generation and python inject.py <system-path> for source-level mutant injection. The implementation requires Python 3.8+, javalang , and a Java JDK that provides javap . The target system must provide a PIT XML report, compiled .class files with debug metadata, and the original Java source files. 2.1. Automated Dataset Generation Dataset generation turns each PIT report entry into a method-level record. For a Java system P with PIT report R_{P} , PITMuS defines the dataset as \mathcal{D}_{P}=\{\,\langle\mathrm{orig}_{i},\mathrm{mut}_{i},\mathrm{doc}_{i},\mathrm{meta}_{i}\rangle\mid m_{i}\in R_{P}\,\}. Each record corresponds to one PIT mutation m_{i} . Here, \mathrm{orig}_{i} is the original enclosing method, \mathrm{mut}_{i} is the mutated version of that method, \mathrm{doc}_{i} is the preceding Javadoc when present, and \mathrm{meta}_{i} stores metadata such as the mutant ID, line number, file path, bytecode index, and mutation description. The main challenge is that a PIT mutation description and line number do not always identify a unique source token. A single source line may contain multiple candidate operators. To resolve this, PITMuS combines the XML report, bytecode information from the compiled class file, and source parsing with javalang ( 11 ) . It first identifies candidate source tokens, then uses bytecode information when needed to select the correct occurrence. After applying the edit, PITMuS extracts the enclosing method and preceding Javadoc when available. Algorithm 1 explains the detailed steps for dataset generation. 2.2. Source-Level Mutant Injection Source-level injection materializes reconstructed mutants as Java

Read the complete text