Artisan: Agentic Artifact Evaluation
Doehyun Baek, Michael Pradel · 2026
Read Artisan: Agentic Artifact Evaluation by Doehyun Baek, Michael Pradel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10046.txt.
Description
Reproducing a paper's numeric tables is essential for building on its results, yet agents that attempt it rarely leave executable proof. A 2026 paper by Doehyun Baek and Michael Pradel presents Artisan, an agent that reproduces tables of numeric results from a paper and its artifact. The task is framed as code generation, so users can audit and rerun the reproduction script themselves.
Automated judges steer the agent toward correct results without revealing them and block shortcuts such as copying precomputed numbers. The team also built Artisan-Bench, sixty tasks drawn from twenty-three software engineering papers. Experiments show Artisan works effectively and efficiently, and its runs surfaced twenty new errors in papers or artifacts.
Catalog record
| CREATOR | Doehyun Baek, Michael Pradel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-10046