RepoDebug : Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
Jingjing Liu, Zeming Liu, Zihao Cheng, Mengliang He, Xiaoming Shi, Yuhang Guo, Xiangrong Zhu, Yuanfang Guo, Yunhong Wang, Haifeng Wang · 2025
Read RepoDebug : Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models by Jingjing Liu, Zeming Liu, Zihao Cheng, Mengliang He, Xiaoming Shi, Yuhang Guo, Xiangrong Zhu, Yuanfang Guo, Yunhong Wang, Haifeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04078.txt.
Description
Jingjing Liu and nine co-authors introduce a harder test for software-fixing language models in a 2025 research paper. Most bug-fixing benchmarks use single small functions. Real projects are whole repositories, meaning many linked files, where errors are messier. Their new data set, RepoDebug, spans eight programming languages and three kinds of debugging work. It also sorts mistakes into twenty-two subtypes, from simple typos to deeper logic faults.
Ten leading language models were tried on the benchmark. Even the strongest one, Claude 3.5 Sonnet, struggled with repository-level repair. The gap shows how far automated debugging still sits from everyday developer work. RepoDebug gives builders a shared yardstick for closing that gap.
Catalog record
| CREATOR | Jingjing Liu, Zeming Liu, Zihao Cheng, Mengliang He, Xiaoming Shi, Yuhang Guo, Xiangrong Zhu, Yuanfang Guo, Yunhong Wang, Haifeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04078