Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab

Haonan DuanUniversity of Toronto, Stephen Zhewen LuSickKidsCaitlin F. HarriganUniversity of TorontoNishkrit DesaiAxiomJiarui LuMilaMichał KoziarskiSickKidsLeonardo CottaVector InstituteChris J. MaddisonUniversity of Toronto · 2025

Read Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab by Haonan DuanUniversity of Toronto, Stephen Zhewen LuSickKidsCaitlin F. HarriganUniversity of TorontoNishkrit DesaiAxiomJiarui LuMilaMichał KoziarskiSickKidsLeonardo CottaVector InstituteChris J. MaddisonUniversity of Toronto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02083.txt.

Description

Duan and co-authors build a dry lab to test whether language models can do real science, in a 2025 benchmark paper. Designing experiments and reading results are core research skills, but wet labs are too costly for routine AI testing. Their benchmark, SciGym, simulates biological systems in a standard markup format. Models must propose perturbations, run simulated experiments, and infer how each system works.

Six frontier models were tested on 137 small systems, with 350 systems released in total. Stronger models did better, yet every model's performance fell steeply as systems grew more complex. The authors read the drop as headroom for growth in machine scientific ability. The benchmark offers a cheap, repeatable arena for measuring that growth.

Catalog record

CREATORHaonan DuanUniversity of Toronto, Stephen Zhewen LuSickKidsCaitlin F. HarriganUniversity of TorontoNishkrit DesaiAxiomJiarui LuMilaMichał KoziarskiSickKidsLeonardo CottaVector InstituteChris J. MaddisonUniversity of Toronto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE145 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02083

Browse the full library · About the Tower of Babel