Automated Benchmark Generation for Repository-Level Coding Tasks

Author, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev · 2025

Read Automated Benchmark Generation for Repository-Level Coding Tasks by Author, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07701.txt.

Description

Konstantinos Vergopoulos, Mark Niklas Müller, and Martin Vechev automate in 2025 the building of coding benchmarks from real software projects. Famous test sets for code agents rest on hand built setups for a dozen popular projects, a narrow base that may skew scores. Their system rebuilds any project's past state on its own.

It sets up old dependencies, runs the test suite, and reads the results without human help. Two fresh data sets came out of the pipeline. Wider project coverage should give fairer measures of what code agents can do. The work targets the quiet bottleneck behind agent progress claims.

Catalog record

CREATORAuthor, Konstantinos Vergopoulos, Mark Niklas Müller, Martin Vechev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07701

Browse the full library · About the Tower of Babel