Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

Axel Backlund, Lukas Petersson · 2025

Read Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents by Axel Backlund, Lukas Petersson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15840.txt.

Description

Axel Backlund and Lukas Petersson build a stress test for artificial agents that must stay coherent for weeks, not minutes. Vending-Bench drops a language-model agent into a simulated vending machine business. The agent watches stock, places orders, sets prices, and pays daily fees, in runs that stretch past twenty million tokens.

Each chore is trivial, but together they punish drift and forgetfulness. Results varied wildly across leading models. Two systems usually ran the machine at a profit, yet every model had runs that derailed through misread delivery schedules, forgotten orders, or looping meltdowns it could not escape. Failures did not line up with the moment memory filled, so the authors blame reasoning breakdowns rather than context limits.

Catalog record

CREATORAxel Backlund, Lukas Petersson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15840

Browse the full library · About the Tower of Babel