Towards a Standard, Enterprise-Relevant Agentic AI Benchmark Lessons from 5.5 billion tokens’ worth of agentic AI evaluations
JV Roig · 2025
Read Towards a Standard, Enterprise-Relevant Agentic AI Benchmark Lessons from 5.5 billion tokens’ worth of agentic AI evaluations by JV Roig (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08042.txt.
Description
J V Roig presents KAMI, a benchmark meant to judge agentic AI systems on work that resembles enterprise use. Ordinary language model tests suffer from training data contamination and ignore multi step tool use under uncertainty. Companies need rankings that predict real deployment value, not just exam style scores.
The index runs one hundred seventy thousand test items, over five and a half billion tokens, across thirty five model setups. Traditional rankings poorly predicted practical agentic results, and newer models did not always beat their predecessors. The report also examines cost against performance, model behavior patterns, and how reasoning affects token efficiency.
Catalog record
| CREATOR | JV Roig |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08042