On Randomness in Agentic Evals
Bjarni Haukur Bjarnason, André Silva, Martin Monperrus · 2026
Read On Randomness in Agentic Evals by Bjarni Haukur Bjarnason, André Silva, Martin Monperrus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07150.txt.
Description
Benchmark scores for AI agents usually come from one run per task, as if that number were stable. Bjarni Haukur Bjarnason and two co-authors test that assumption with 60,000 agent trajectories on SWE-Bench-Verified, across three models and two scaffolds.
Single-run scores swing by 2. 2 to 6. 0 percentage points depending on which run is picked, even at temperature zero. That means a reported gain of two or three points may be noise rather than progress. Tracing tokens shows runs splitting early and cascading into other strategies. The paper argues for repeated runs and honest error bars in agent testing.
Catalog record
| CREATOR | Bjarni Haukur Bjarnason, André Silva, Martin Monperrus |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-07150