Agentic Bug Reproduction for Effective Automated Program Repair at Google
Runxiang Cheng 1∗ , Michele Tufano 2 , Jürgen Cito 3∗ , José Cambronero 2 Pat Rondon 2 , Renyao Wei 2 , Aaron Sun 2 , Satish Chandra 2 · 2025
Read Agentic Bug Reproduction for Effective Automated Program Repair at Google by Runxiang Cheng 1∗ , Michele Tufano 2 , Jürgen Cito 3∗ , José Cambronero 2 Pat Rondon 2 , Renyao Wei 2 , Aaron Sun 2 , Satish Chandra 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01821.txt.
Introduction
Runxiang Cheng and seven co-authors automate bug reproduction at Google in twenty twenty-five. Bug reports rarely include enough detail to reproduce a defect. A reproduction test that fails before the fix and passes after is gold for debugging. Such tests are almost never attached to reports, in open source or industry.
The team works with real bugs from Google's internal tracker. Generating reproduction tests in a huge proprietary codebase is the challenge. They adapt a state-of-the-art technique called LIBRO to the setting. Their own agent-based system uses a fine-tuned language model to write the tests. Readers in software engineering get evidence on automated program repair where the code base is industrial scale and the bugs are real.
Catalog record
| CREATOR | Runxiang Cheng 1∗ , Michele Tufano 2 , Jürgen Cito 3∗ , José Cambronero 2 Pat Rondon 2 , Renyao Wei 2 , Aaron Sun 2 , Satish Chandra 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01821
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentic Bug Reproduction for Effective Automated Program Repair at Google Thanks: *Runxiang Cheng and Jürgen Cito conducted this research at Google. Runxiang Cheng 1∗ , Michele Tufano 2 , Jürgen Cito 3∗ , José Cambronero 2 Pat Rondon 2 , Renyao Wei 2 , Aaron Sun 2 , Satish Chandra 2 Abstract . Bug reports often lack sufficient detail for developers to reproduce and fix the underlying defects. Bug Reproduction Tests (BRTs), tests that fail when the bug is present and pass when it has been resolved, are crucial for debugging, but they are rarely included in bug reports, both in open-source and in industrial settings. Thus, automatically generating BRTs from bug reports has the potential to accelerate the debugging process and lower time to repair. This paper investigates automated BRT generation within an industry setting, specifically at Google, focusing on the challenges of a large-scale, proprietary codebase and considering real-world industry bugs extracted from Google’s internal issue tracker. We adapt and evaluate a state-of-the-art BRT generation technique, LIBRO, and present our agent-based approach, BRT Agent, which makes use of a fine-tuned Large Language Model (LLM) for code editing. Our BRT Agent significantly outperforms LIBRO, achieving a 28% plausible BRT generation rate, compared to 10% by LIBRO, on 80 human-reported bugs from Google’s internal issue tracker. We further investigate the practical value of generated BRTs by integrating them with an Automated Program Repair (APR) system at Google. Our results show that providing BRTs to the APR system results in 30% more bugs with plausible fixes. Additionally, we introduce Ensemble Pass Rate (EPR), a metric which leverages the generated BRTs to select the most promising fixes from all fixes generated by APR system. Our evaluation on EPR for Top-K and threshold-based fix selections demonstrates promising results and trade-offs. For example, EPR correctly selects a plausible fix from a pool of 20 candidates in 70% of cases, based on its top-1 ranking. Proceedings of ACM Conference (Conference’17) CCS: Software and its engineering Software testing and debugging 1. Introduction Bug reproduction is a critical part of the software debugging process ( Chaparro et al., 2019 ; Kang et al., 2023b ; Koyuncu et al., 2019 ; Just et al., 2014 ; Le Goues et al., 2012 ; Xiong et al., 2017 ; Li et al., 2019 ; Yang et al., 2024a ) . The bug reproduction steps of a reported bug can often be encapsulated into test(s), which automatically reproduce the bug by demonstrating test failure(s). Such a test is referred to as a Bug Reproduction Test (BRT). More formally, a BRT should (1) fail in the presence of the bug, demonstrating the faulty behavior; and (2) pass once the bug has been resolved, confirming the effectiveness of the applied fix patch. For both human users and Automated Program Repair (APR) systems, BRTs can offer insights on the root cause of the bug ( Ko and Myers, 2008 ; Kang et al., 2023b ; Nayrolles et al., 2015 ; Just et al., 2018 ) , guide the development of a fix patch ( Rondon et al., 2025 ; Yang et al., 2024a ; Mündler et al., 2024 ) , and is essential for validating the effectiveness of the proposed fix patch ( Ko and Myers, 2008 ; Just et al., 2014 ; Saha et al., 2017 ; Le Goues et al., 2012 ) . However, despite their importance, BRTs are often absent when the debugging or repairing process starts. Bug reports from human users/developers, written in natural language, are often vague, ambiguous, and lacking crucial details for reproducing and subsequently addressing the reported software defects ( Bettenburg et al., 2008 ; Zimmermann et al., 2009 ; Anvik et al., 2006 ) . Studies show that bug reports in open-source projects often do not have BRTs, e.g., Koyuncu et al. (2019) found that only 4% of the bug reports in Defects4J ( Just et al., 2014 ) include a BRT; Kang et al. (2023b) found that 28% of the tests among the 300 most-starred GitHub projects are added due to bug reports; more recently, Mündler et al. (2024) found that there are no corresponding BRTs for bug-reporting pull requests prior to the creation of those pull requests in SWE-Bench projects. In industrial setting, similarly, BRTs are commonly expected to be developed alongside the fix patch, after the bug is reported. Through our exploration of BRT creation and BRT availability in Google’s internal issue tracking system, we observed that BRT development can be deferred to fix development stage because developers often do not prioritize the task of, or possess enough information of, developing a BRT at the time of bug reporting (§ 2 ). For example, developers may not attend to BRT development when actively dealing with issues reported by production monitoring. Moreover, developing a BRT requires knowledge of the bug’s root cause, and effort to craft a precise, bug-reproducing test failure ( Grottke and Trivedi, 2007 ; Just et al., 2018 ; Beller et al., 2018 ) , which may not be available as soon as the bug is reported. In this paper, we tackle the challenge of automated BRT generation in an industrial environment, specifically at Google. Automated BRT generation facilitates BRT development at scale, which could provide crucial benefits to industrial settings where a large number of bug reports are handled per day. To develop automated BRT generation for Google’s development environment, we first make nontrivial effort to adapt a state-of-the-art LLM-based approach, LIBRO ( Kang et al., 2023b ) (§ 4.1 ). After finding limitations from the adapted LIBRO, we developed our LLM-agent-based approach for BRT generation, BRT Agent (§ 4.2 ). Our approach, BRT Agent, goes beyond being a straightforward adaptation of the existing LLM-agent-based APR systems into BRT generation task via LLM prompt engineering (§ 3 ). Specifically, we introduce and focus on agent components that overcome challenges unique to our industrial context. For example, to address the challenge of generating proprietary code with idiosyncratic APIs and coding practices, BRT Agent incorporates a specialized code-editing LLM fine-tuned on Google’s codebase, to generate high-quality test code (§ 4.2.3 ). We implemented both LIBRO and our BRT Agent to work with Google’s development infrastructure, and evaluate them on a set of 80 production bugs reported and fixed by human developers from a wide array of Google projects ( Rondon et al., 2025 ) (§ 5.1 ). Moreover, while prior work focuses on evaluating the performance of BRT generation techniques in isolation (§ 3 ), we further investigate the practical impact of the generated BRTs for bug-fixing. Given BRTs can provide additional context for understanding the bug, we study how generated BRTs can improve the performance of Passerine ( Rondon et al., 2025 ) , an industrial-scale APR system at Google, to generate plausible fixes for more bugs more efficiently. Given APR systems can generate many fixes per bug and BRTs can help validate a fix through Fail-to-Pass behavior (§ 2 ), we explore whether generated BRTs can be used to effectively indicate the plausible fixes from all fixes generated by Passerine . To this aim, we propose a novel metric, Ensemble Pass Rate (EPR), which evaluates each generated fix by calculating