Automated Benchmark Generation for Repository-Level Coding Tasks
Author, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev · 2025
Read Automated Benchmark Generation for Repository-Level Coding Tasks by Author, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07701.txt.
Description
Konstantinos Vergopoulos, Mark Niklas Müller, and Martin Vechev automate in 2025 the building of coding benchmarks from real software projects. Famous test sets for code agents rest on hand built setups for a dozen popular projects, a narrow base that may skew scores. Their system rebuilds any project's past state on its own.
It sets up old dependencies, runs the test suite, and reads the results without human help. Two fresh data sets came out of the pipeline. Wider project coverage should give fairer measures of what code agents can do. The work targets the quiet bottleneck behind agent progress claims.
Catalog record
| CREATOR | Author, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07701