What’s in a Benchmark? The Case of SWE-Bench in Automated Program Repair Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP ’26), April 12–18, 2026,
Matias Martinez, Xavier Franch · 2026
Read What’s in a Benchmark? The Case of SWE-Bench in Automated Program Repair Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP ’26), April 12–18, 2026, by Matias Martinez, Xavier Franch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04449.txt.
Description
Who actually leads the rankings on a famous coding benchmark? Matias Martinez and Xavier Franch examine that question in a 2026 study of SWE-Bench, a test that asks repair tools to fix real issues from open-source Python projects.
The authors analyzed entries on the Lite and Verified leaderboards. They recorded who submitted each entry, what product stood behind it, and which language model it used. Most entries came from industry, especially small firms and large public companies. Proprietary models, above all the Claude family, dominated the top results. Still, academic entries, usually open source, remained competitive. The study urges more transparency and diversity in benchmark research.
Catalog record
| CREATOR | Matias Martinez, Xavier Franch |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04449