Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Wang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia, [0.25em] University of Southern California · 2026

Read Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? by Wang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia, [0.25em] University of Southern California (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17338.txt.

Introduction

Chai and seven colleagues ask whether code-debugging models truly fix bugs or simply rewrite whole programs. Their Precise Debugging Benchmark turns any coding dataset into a debugging test by planting verified single bugs and combining them into multi-bug programs. Two new scores track how many edits were truly needed and how many bugs were actually resolved. Leading models passed over 76 percent of unit tests yet scored below 45 percent on precision, even when told to edit minimally, and fancier iterative strategies barely helped.

Catalog record

CREATORWang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia, [0.25em] University of Southern California
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59804
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? Wang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia, [0.25em] University of Southern California Abstract Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmarking ( PDB ) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall , which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76 % 76\% but exhibit precision below 45 % 45\% , even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models. † † ∗ Equal contribution. † † † Work done before joining Microsoft. Figure 1: Real example from GPT-5.2 debugging a binary search program, where the model rewrites the entire solution. Green lines mark precise edits; gray lines highlight over-edits. 1 Introduction Large Language Models (LLMs) have reshaped the programming landscape through their remarkable capabilities in code generation ( 5 ; 23 ) . From synthesizing complex algorithms from natural language prompts to translating entire codebases, modern LLMs excel at producing code from scratch. However, real-world software development is dominated not by generation but by debugging and maintenance ( 11 ) . When applied to debugging tasks, we observe that frontier LLMs often default to regeneration, i.e . , rewriting large portions, or even the entirety, of a program when presented with buggy code (Figure 1 ). While often effective at passing tests, this brute-force strategy is poorly suited for realistic codebases, where large-scale rewrites are costly, risky, and difficult to review ( 39 ) . In contrast, targeted debugging requires precise fault localization and minimal, intent-preserving edits. This raises a fundamental question: How far are LLMs from precise debugging, rather than merely reverting to their strength in code regeneration? Existing debugging benchmarks focus on unit-test only evaluation and fail to evaluate these capabilities. Under such evaluation, models are rewarded equally for regenerating a full solution, hard-coding outputs, or performing a minimal targeted fix. Moreover, unit-test evaluation obscures incremental progress: a model that correctly repairs only one defect in a multi-bug program receives the same score as a model that fixes none. This misalignment with real-world debugging practice limits our ability to understand how LLMs reason about bugs and code edits. To address this gap, we introduce the Precise Debugging Benchmarking ( PDB ) framework, an automatic pipeline that rigorously evaluates LLM debugging behavior independently of code generation. PDB provides a plug-and-play framework that converts existing coding datasets into debugging benchmarks through two steps: (1) synthesizing verified atomic bugs to produce ground-truth edit scripts, and (2) composing these bugs into multi-bug programs while preserving bug independence ( i.e . , avoiding compounding interactions). Beyond binary test outcomes, PDB evaluates model patches using novel edit-level precision and bug-level recall , explicitly rewarding targeted fixes and penalizing unnecessary modifications. We constructed a 5,751-example PDB-Single-Hard and a 256-example PDB-Multi evaluation benchmarks with the PDB framework, from BigCodeBench ( 55 ) and LiveCodeBench ( 54 ) . Experiments on PDB-Single-Hard reveal behaviors that unit tests fail to capture. First, frontier models exhibit strikingly different rankings under edit-level evaluation . Models such as GPT-5.1-Codex 31 and DeepSeek-V3.2-Thinking 25 achieve high unit-test pass rates ( > 76\% ) but low edit precision ( \leq 45\% ), while Qwen3-Coder-480B 34 attains comparatively lower unit-test pass rates ( 70\% ) yet substantially higher precision ( 66\% ). This ranking inversion persists on the multi-line benchmark PDB-Multi , indicating that the precision gap reflects a consistent tendency toward regeneration. Additionally, we show that though iterative and agentic debugging strategies can improve unit-test performance, they do not meaningfully improve precision or recall. Our findings demonstrate the necessity of PDB for revealing true debugging capabilities beyond surface-level correctness, and highlight a fundamental limitation in current post-training pipelines for coding LLMs. Figure 2: PDB pipeline. Generation: LLMs first synthesize and verify single-line bugs from existing coding datasets, which are then composed into multi-bug programs. Evaluation: Automated debugging systems are evaluated on these programs using both unit-test accuracy and edit-level precision and bug-level recall. 2 Precise Debugging Setup We begin by formally defining the components of the automated debugging task. An automated debugging system \mathcal{M} , takes the initial buggy program C_{\text{b}} and a natural language task description x as input, and returns the predicted program revision \hat{C}=\mathcal{M}(C_{\text{b}},x) . The conventional debugging pipeline evaluates the system’s final output purely on its functional correctness, using a binary evaluation function F_{{\mathcal{U}}}(C)\rightarrow\{0,1\} , where {\mathcal{U}}=\{u_{1},u_{2},\dots,u_{n}\} is a suite of designed unit tests. The evaluation function F_{{\mathcal{U}}} returns 1 if the program C passes all tests in {\mathcal{U}} , and 0 otherwise. While straightforward, this method cannot penalize unnecessary edits or wholesale rewrites when the final program merely passes the test, nor can it distinguish between partially correct solutions and entirely incorrect ones. The precise debugging setup shifts the evaluation from program-level to the specific set of edits proposed by the model. Minimal corrections. We denote a line-edit on line l as e_{l} , and a set of line-edits as E . For a buggy program C_{\text{b}} , we denote the set of minimal corrections by {\mathcal{E}}_{C_{\text{b}}}=\argmin_{E}|E|\ \text{s.t.}\ F_{{\mathcal{U}}}(\operatorname{apply}(E,C_{\text{b}}))=1, where the \operatorname{apply} function applies line-edits on C_{\text{b}} . Similarly, we can apply reverse edits \bar{E} on ground-truth program C_{\text{gt}} to derive buggy program C_{\text{b}} . Atomicity. We define the bug in a buggy program C_{\text{b}} as atomic when a minimal correction consists of edits on a contiguous sequence of lines. Formally, \exists E\in{\mathcal{E}}_{C_{\text{b}}} such that E=\{e_{i},e_{i+1},\dots,e_{i+n}\}. Independence. Intuitively, independence means that fixing one bug neither introduces nor removes edits required to fix the other. For two edit sets E_{1}\in{\mathcal{E}}_{C_{\text{b1}}} and E_{2}\in{\mathcal{E}}_{C_{\text{b2}}} corresponding to the same ground-truth program C_{\text{gt}} , we can construct a composed buggy program C_{\text{b3}}=\operatorname{apply}(\bar{E}_{1}\cup\bar{E}_{2},C_{\text{gt}}). If the set of minimal corrections is the pairwise union of corrections from {\mathcal{E}}_{C_{\text{b1}}} and {\mathcal{E}}_{C_{\text{b2}}} , we consider bugs in C_{\text{b1}} and C_{\text{b2}} to be independent . Semantic correctness. Consider a buggy program C_{\text{b}} containing k atomic and independent bugs, and a revision \hat{C}=\operatorname{apply}(\hat{E},C_{\text{b}}) , where \hat{E} is the predicted edits. We define bug-level semantic correctness as follows. Let E_{\text{gt}}\in{\mathcal{E}}_{C_{\text{b}}} be the set of ground-truth edits, which can be decomposed as E_{\text{gt}}=E_{1}\cup E_{2}\cup\dots\cup E_{k} , where E_{1},\dotsc,E_{k} are contiguous and non-overlapping. We employ a function, denoted as \operatorname{map} , which pairs each E_{i} with the closest edits in \hat{E} . For each bug i , we construct a pseudo-revision \hat{C}_{i}=\operatorname{apply}((E_{\text{gt}}\setminus E_{i})\cup\operatorname{map}(E_{i}),\,C_{\text{b}}) , which replaces the ground-truth edits E_{i} with the predicted edits \operatorname{map}(E_{i}) . We define a candidate \hat{C}_{i} as semantically correct for bug i if F_{{\mathcal{U}}}(\hat{C}_{i})=1

Read the complete text