Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques
David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro · 2026
Read Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques by David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11262.txt.
Description
David Williams and coauthors benchmark the tools meant to catch overfitted patches from automated program repair. Such patches pass the tests yet are still wrong. They find the detectors wanting. Earlier evaluations used unrealistic mixes of correct and overfitting patches. The team curated datasets matching what repair tools produce in use. They tested six detectors against random sampling baselines. Simple random selection beat every detector in most cases. This striking result argues that two things are overdue. New techniques are overdue. Realistic benchmarking against randomness is overdue too.
Catalog record
| CREATOR | David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67606 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11262