Artisan: Agentic Artifact Evaluation

Doehyun Baek, Michael Pradel · 2026

Read Artisan: Agentic Artifact Evaluation by Doehyun Baek, Michael Pradel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10046.txt.

Description

Reproducing a paper's numeric tables is essential for building on its results, yet agents that attempt it rarely leave executable proof. A 2026 paper by Doehyun Baek and Michael Pradel presents Artisan, an agent that reproduces tables of numeric results from a paper and its artifact. The task is framed as code generation, so users can audit and rerun the reproduction script themselves.

Automated judges steer the agent toward correct results without revealing them and block shortcuts such as copying precomputed numbers. The team also built Artisan-Bench, sixty tasks drawn from twenty-three software engineering papers. Experiments show Artisan works effectively and efficiently, and its runs surfaced twenty new errors in papers or artifacts.

Catalog record

CREATORDoehyun Baek, Michael Pradel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10046

Browse the full library · About the Tower of Babel