AutomationBench

Daniel Shepard, Robin Salimans · 2026

Read AutomationBench by Daniel Shepard, Robin Salimans (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18934.txt.

Description

Shepard and Salimans introduce AutomationBench, a benchmark for AI agents on cross application workflows. Real workflows demand coordination, API discovery, and policy adherence across a CRM, inbox, calendar, and messaging platform. Tasks span Sales, Marketing, Operations, Support, Finance, and HR. Agents must find endpoints alone, follow business rules, and navigate misleading records. Grading is automated and end state only, and even the best models score below 10 percent.

Catalog record

CREATORDaniel Shepard, Robin Salimans
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18698
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18934

Browse the full library · About the Tower of Babel