HeurekaBench : A Benchmarking Framework for AI Co-scientist

Siba Smarak Panigrahi, Jovana Videnović, Maria BrbićÉcole Polytechnique Fédérale de Lausanne (EPFL) brbiclab.epfl.ch/projects/heurekabench · 2026

Read HeurekaBench : A Benchmarking Framework for AI Co-scientist by Siba Smarak Panigrahi, Jovana Videnović, Maria BrbićÉcole Polytechnique Fédérale de Lausanne (EPFL) brbiclab.epfl.ch/projects/heurekabench (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01678.txt.

Description

AI systems that act as research assistants can now run multi-step analyses, but testing whether their science is any good has lagged behind. Siba Smarak Panigrahi, Jovana Videnović, and Maria Brbić introduce HeurekaBench, a framework that turns real published studies and their code into open-ended benchmark questions for these AI co-scientists.

Questions are drafted with help from several language models, then checked against the findings the original studies reported. The team applies the framework to single-cell biology and compares current agents on the resulting benchmark. Adding a critic module improved weak answers from open-source agents by up to twenty-two percent. That nearly closed the gap with closed systems. The benchmark also exposes how design choices affect agent quality.

Catalog record

CREATORSiba Smarak Panigrahi, Jovana Videnović, Maria BrbićÉcole Polytechnique Fédérale de Lausanne (EPFL) brbiclab.epfl.ch/projects/heurekabench
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01678

Browse the full library · About the Tower of Babel