Giulio Starace

* · 2025

Read Giulio Starace by * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01848.txt.

Description

Giulio Starace and co-authors ask whether AI agents can redo published AI research from scratch, in a 2025 paper. Their benchmark, PaperBench, hands agents twenty spotlight and oral papers from a top 2024 conference. The agent must grasp the contribution, write the code, and run the experiments, with no starter files. Grading uses rubrics that break each paper into small checkable pieces, over eight thousand in total, written with the original authors.

An automated judge based on a language model grades attempts against the rubrics, and the judge itself was benchmarked. Replication is the sternest test of whether an AI understands research rather than summarizing it. The results map how far today's agents get on real science.

Catalog record

CREATOR*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01848

Browse the full library · About the Tower of Babel