Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Modelling and simulation

Hubert M. Pysklo, Artem Zhuravel, Patrick D. Watson · 2026

Read Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Modelling and simulation by Hubert M. Pysklo, Artem Zhuravel, Patrick D. Watson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11224.txt.

Description

Agent benchmarks must balance control against realism. Hubert M. Pysklo and two co-authors present Agent-Diff, a frame for testing language-model agents on enterprise API tasks through code execution. It keeps real API interfaces while sandboxing the setting where calls run and are judged.

Two ideas drive it: a state-diff contract that defines success by the change in setting state, not by trace matching, and a sandbox of containerized replicas of enterprise APIs. Nine models were tested across hundreds of enterprise workflow tasks, with ablations on the value of API documentation. The paper offers a controlled yet realistic yardstick for agentic productivity software.

Catalog record

CREATORHubert M. Pysklo, Artem Zhuravel, Patrick D. Watson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE232 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-11224

Browse the full library · About the Tower of Babel