SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava, Valerie Chen, Debojyoti Dutta · 2026
Read SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories by Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava, Valerie Chen, Debojyoti Dutta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13713.txt.
Introduction
Aditya Bharat Soni and co-authors train open models to write tests that replay reported bugs in 2026. A good reproduction test shows a bug in code from a plain-language issue report. Such tests help devs find the root cause fast. They also back test-first work, where the test comes before the fix.
Their system, SWE-Tester, trains open-source language models for this task. It works on real-world code stores, not toy examples. Better bug replay tests can also make coding agents more effective. The work targets a slow, manual step in daily software work.
Catalog record
| CREATOR | Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava, Valerie Chen, Debojyoti Dutta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13713
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava, Valerie Chen, Debojyoti Dutta Abstract Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development – “test first, write code later”, and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester – a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10% in success rate and 21% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain. 1 Introduction Large language models have demonstrated remarkable improvements for software engineering (SWE) tasks which is evident from progress on benchmarks like SWE-Bench ( 15 ) which evaluates an LLM’s ability to resolve issues in GitHub repositories. However, issue resolution is only one of the many responsibilities of a software developer which also includes writing tests, reviewing pull requests (PRs), developing system design etc. Although rigorous software testing is a critical component of the software development lifecycle, a large fraction of tests in repositories are added after issues are reported ( 17 ) , and reproduction tests are rarely available in the test suite when an issue is reported ( 17 ; 20 ) . Despite its importance, developers often dislike writing tests and consider it a mundane task ( 26 ) . Given these observations, automating the generation of issue reproduction tests can significantly enhance developer productivity and facilitate test-driven development (TDD) which is known to improve code and test quality by encouraging tests to be written before source code modifications ( 8 ) . Furthermore, reproduction tests are also leveraged by automated issue resolution approaches to rerank candidate bug fixes sampled from the LLM ( 33 ; 32 ) and leverage inference-time compute using execution-based verifiers ( 14 ) . Prior work in this domain has proposed evaluation benchmarks like SWT-Bench ( 20 ) and Defects4J ( 16 ) , and has developed various scaffolds for automated generation of issue reproduction tests ( 1 ; 21 ; 17 ; 30 ; 18 ) . Although effective, a crucial limitation of existing methods is their reliance on proprietary closed-source LLMs to achieve strong performance. While various efforts have been made to train LLMs for fixing issues by editing source code ( 14 ; 23 ; 37 ; 34 ) , training LLMs for reproducing issues by writing reproduction tests is an unexplored research direction. Figure 1: Our proposed SWE-Tester pipeline. Given an issue description and the pre-PR repository (which has this issue/bug), we first retrieve defective source code and relevant test file (code localization). Next, we edit the test file to augment it with reproduction tests (code editing). Furthermore, while issue resolution and issue reproduction are related tasks, both involving repository-level code generation, 20 observe that there is no correlation between an LLM’s ability to fix an issue and reproduce it, and conclude that they are distinct tasks of different difficulty. To address this research gap, we propose SWE-Tester – a novel framework for generating issue reproduction tests using fully open-source LLMs. As shown in Figure 1 , we follow a standard two-step workflow consisting of code localization followed by code editing. Our analysis reveals that while open-source LLMs have reasonable code localization performance, they significantly struggle with code editing. Since code editing is the primary bottleneck in the workflow, we particularly focus on training LLMs to improve their code editing performance. We summarize our research contributions below: 1. Training dataset for issue reproduction test generation : We leverage raw PR data from open-source GitHub repositories to curate a 41K training instances from 2.6K repositories. 2. Training LLMs using our dataset improves their ability to write comprehensive issue reproduction tests : Our training recipe generalizes to five open-source LLMs of varying sizes and model families, Qwen-2.5 Coder Instruct (7B, 14B and 32B), Llama3.1-8B Instruct and Gemma3-12B Instruct, with absolute gains of upto 10% in success rate and upto 21% in change coverage on SWT-Bench Verified. 3. Scaling inference-time compute and training data consistently improves model performance : Our experiments reveal that model performance improves as we increase training data and inference-time compute and provide crucial insights for future improvements. 2 Related Work Benchmarks and Training Methods for Issue Reproduction Test Generation. Reproduction test generation is more challenging than unit test generation because instead of writing tests from existing code, tests must be derived from a natural language issue description and a buggy codebase, without access to the issue-fixing patch. Evaluation benchmarks for evaluating the ability of LLMs to write reproduction tests include SWT-Bench ( 20 ) for Python repositories and Defects4J ( 16 ) for Java codebases. While SWT-Bench uses unit test mode by default, where the LLM-generated tests must be integrated into repository’s existing test suite, it also supports an easier reproduction script mode where the LLM generates a stand-alone Python script for issue reproduction. Prior training approaches are limited to the reproduction script mode. 14 train LLMs as a testing agent using rejection sampling fine-tuning ( 4 ) on trajectories sampled from expensive closed-source LLMs, while 11 use reinforcement learning for a pipeline-based workflow, with limited performance gains due to unstable learning from sparse rewards. Category Metric Dataset Size # Instances 41K # Repos 2.6K Issue Text # words 175.7 Gold code patch # Files edited 1.4 # Lines edited 37.2 Gold test patch # Files edited 1 # Lines edited 47.9 Table 1: Dataset statistics for our training dataset (left) and histogram of repo-level instance frequencies (right). Except for size, we report the instance-level averages for all metrics in the table. The reproduction script mode is unrealistic since developer-written PRs typically modify the existing test suite, making training data curation difficult for this mode. We train LLMs for unit test mode by leveraging developer-written tests in issue-resolving PRs. Workflows for Reproduction Test Generation Prior approaches have proposed specialized pipeline-based workflows ( 1 ; 17 ; 18 ; 21 ) that often prompt a closed-source LLM for various steps like task planning, code localization, root cause analysis, iterative test refinement, etc., and have developed LLM agents like AEGIS ( 30 ) and SWE-Agent+ ( 20 ) . Furthermore, 20 also issue resolution systems like 5 ; 39 for this task. While effective, they often rely on closed-source LLMs for strong performance. Furthermore, these workflows are not suitable for training LLMs. For pipeline-based methods, there is no ground truth data available for most steps (e.g. planning, root cause analysis), which makes training data curation difficult. Similar problems arise when training LLMs for agent-based workflows since we need