On Randomness in Agentic Evals

Bjarni Haukur Bjarnason, André Silva, Martin Monperrus · 2026

Read On Randomness in Agentic Evals by Bjarni Haukur Bjarnason, André Silva, Martin Monperrus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07150.txt.

Description

Benchmark scores for AI agents usually come from one run per task, as if that number were stable. Bjarni Haukur Bjarnason and two co-authors test that assumption with 60,000 agent trajectories on SWE-Bench-Verified, across three models and two scaffolds.

Single-run scores swing by 2. 2 to 6. 0 percentage points depending on which run is picked, even at temperature zero. That means a reported gain of two or three points may be noise rather than progress. Tracing tokens shows runs splitting early and cascading into other strategies. The paper argues for repeated runs and honest error bars in agent testing.

Catalog record

CREATORBjarni Haukur Bjarnason, André Silva, Martin Monperrus
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07150

Browse the full library · About the Tower of Babel