Towards a Standard, Enterprise-Relevant Agentic AI Benchmark Lessons from 5.5 billion tokens’ worth of agentic AI evaluations

JV Roig · 2025

Read Towards a Standard, Enterprise-Relevant Agentic AI Benchmark Lessons from 5.5 billion tokens’ worth of agentic AI evaluations by JV Roig (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08042.txt.

Description

J V Roig presents KAMI, a benchmark meant to judge agentic AI systems on work that resembles enterprise use. Ordinary language model tests suffer from training data contamination and ignore multi step tool use under uncertainty. Companies need rankings that predict real deployment value, not just exam style scores.

The index runs one hundred seventy thousand test items, over five and a half billion tokens, across thirty five model setups. Traditional rankings poorly predicted practical agentic results, and newer models did not always beat their predecessors. The report also examines cost against performance, model behavior patterns, and how reasoning affects token efficiency.

Catalog record

CREATORJV Roig
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08042

Browse the full library · About the Tower of Babel