Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
Axel Backlund, Lukas Petersson · 2025
Read Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents by Axel Backlund, Lukas Petersson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15840.txt.
Description
Axel Backlund and Lukas Petersson build a stress test for artificial agents that must stay coherent for weeks, not minutes. Vending-Bench drops a language-model agent into a simulated vending machine business. The agent watches stock, places orders, sets prices, and pays daily fees, in runs that stretch past twenty million tokens.
Each chore is trivial, but together they punish drift and forgetfulness. Results varied wildly across leading models. Two systems usually ran the machine at a profit, yet every model had runs that derailed through misread delivery schedules, forgotten orders, or looping meltdowns it could not escape. Failures did not line up with the moment memory filled, so the authors blame reasoning breakdowns rather than context limits.
Catalog record
| CREATOR | Axel Backlund, Lukas Petersson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15840