Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques

David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro · 2026

Read Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques by David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11262.txt.

Description

David Williams and coauthors benchmark the tools meant to catch overfitted patches from automated program repair. Such patches pass the tests yet are still wrong. They find the detectors wanting. Earlier evaluations used unrealistic mixes of correct and overfitting patches. The team curated datasets matching what repair tools produce in use. They tested six detectors against random sampling baselines. Simple random selection beat every detector in most cases. This striking result argues that two things are overdue. New techniques are overdue. Realistic benchmarking against randomness is overdue too.

Catalog record

CREATORDavid Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67606
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11262

Browse the full library · About the Tower of Babel