CUBE: A Standard for Unifying Agent Benchmarks
2026
Read CUBE: A Standard for Unifying Agent Benchmarks (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15798.txt.
Introduction
Lacoste and co-authors propose CUBE, a shared standard for agent benchmarks. Each new benchmark now demands bespoke wiring into test platforms. That integration tax slows full evaluation across the field. CUBE wraps a benchmark once so any platform can run it. The protocol builds on MCP and Gym ideas. Task, benchmark, package, and registry layers stay apart in the design. Compliant tools can serve evaluation, training, or data generation. The team invites community help before platform forks harden.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42755 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CUBE: A Standard for Unifying Agent Benchmarks Abstract The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating an “integration tax” that limits comprehensive evaluation. We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym that allows benchmarks to be wrapped once and used everywhere . By separating task, benchmark, package, and registry concerns into distinct API layers, CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration. We call on the community to contribute to the development of this standard before platform-specific implementations deepen fragmentation as benchmark production accelerates through 2026. Alexandre Lacoste 1 Nicolas Gontier 1 Oleh Shliazhko 1 Aman Jaiswal 1,3 Kusha Sareen 1,6,7 Shailesh Nanisetty 1 Joan Cabezas Manuel Del Verme 2 Omar G. Younis 2 Simone Baratta 2 Matteo Avalle 2 Imene Kerboua 7 Xing Han Lù 6,7 Elron Bandel 4 Michal Shmueli-Scheuer 4 Asaf Yehudai 4 Leshem Choshen 4 Jonathan Lebensold 5 Sean Hughes 1 Massimo Caccia 1 Alexandre Drouin 1,7 Siva Reddy 6,7 Tao Yu 8 Yu Su 9 Graham Neubig 10 Dawn Song 11 1 ServiceNow AI Research 2 Silverstream.ai 3 Dalhousie 4 IBM Research 5 Jetty 6 McGill 7 Mila 8 HKU 9 OSU 10 CMU 11 UC Berkeley 1 Introduction The field of Artificial Intelligence (AI) is experiencing a remarkable surge in benchmark development for AI agents. The research community has created an impressive ecosystem of complex, interactive environments designed to test the limits of autonomous agents ( 42 ; 27 ) . This diversity presents tremendous opportunities: researchers can now evaluate agents across broad task distributions and leverage large-scale training on varied environments. However, realizing this potential requires solving a critical integration challenge. The effort required to incorporate these diverse benchmarks into evaluation and training pipelines has become a significant bottleneck. In practice, this limits which labs can afford to evaluate across many benchmarks, and shapes what research gets done as a result ( 2 ) . As we move toward generalist agents, the need for a general framework supporting both evaluation and post-training on diverse task distributions becomes critical. We seek agents that reason, use tools ( 40 ; 32 ) , and navigate unseen environments. With proper tools and a consistent interface, a generalist agent should handle any benchmark ( 4 ) . Yet integrating new benchmarks remains time-consuming, forcing researchers to act more like systems engineers than AI scientists. To address this integration tax, several platforms have emerged, such as NeMo Gym ( 28 ) , Harbor ( 31 ) , HAL ( 13 ) , and OpenEnv ( 25 ) , among others. These platforms integrate existing benchmarks and provide tooling to author new environments, but each proposes its own environment interface. As a result, maintainers must build connectors to move benchmarks across them, a growing burden as benchmark production accelerates. Recognizing this, the community has also begun proposing open benchmark interface standards: the Agentified Agent Assessment (AAA) paradigm in AgentBeats ( 1 ) and CUBE (this paper) are concurrent efforts sharing this motivation, though differing in scope and design choices, as discussed in Section 4 . Competition between platforms is a valuable market force and drives innovation. They should compete on features, usability, scalability, and other value metrics. Yet without a common benchmark interface, competition skews toward the size of their integration catalog instead. The Core Position The community needs a standard that allows practitioners to wrap agentic benchmarks once and have it work everywhere, for evaluation and for training, at scale. Any platform that implements the standard would instantly gain access to every compliant benchmark. This would eliminate redundant integration work and let us focus on understanding and improving agent behavior. By moving toward a universal standard, we can turn the current landscape of isolated silos into a thriving, interoperable ecosystem. The importance of multi-benchmarking cannot be overstated. There are currently over 300 agentic benchmarks available, many of which are highly innovative but remain largely unknown because they are too difficult to set up. By simplifying and scaling cross-benchmark evaluation, we can gain a much deeper understanding of agent capabilities across a vast distribution of tasks. Given the rapid rise of coding agents ( 41 ; 37 ; 12 ; 9 ) and the increasing interest in post-training and RL on diverse tasks ( 14 ; 29 ) , we forecast this number to double by the end of 2026. Without a standard, the field is destined for unmanageable fragmentation as benchmark production outstrips integration capacity. 2 The Current Landscape and its Challenges Table 1 : A comparative analysis of the infrastructure and operational requirements across current agentic benchmarks. The lack of uniformity in how these environments are hosted, controlled, and reset creates a significant overhead for cross-benchmark evaluation. Feature WebArena ( 43 ) SWE-bench ( 12 ) osworld ( 39 ) GAIA ( 26 ) Environment Type Simulated Web : A private micro-internet (GitLab, Reddit clone, etc.). Coding Workspace : A specific software repository and dev toolchain. Desktop OS : A full graphical operating system (Ubuntu/Windows). The Real World : Live internet access and local document sets. Hosting Format Benchmark-level VM : One shared world per task run. High AWS costs. Task-level Container : Lightweight, ephemeral Docker images per task. Benchmark-level VM : OS VM with Task-level Snapshots. Static Files : Task files on HF Hub; users provide tool implementations. Action Space (Tooling) Flexible/Provided : Playwright basics provided, but usually swapped for custom abstractions (e.g., BrowserGym). Fixed Shell : Standard terminal-based actions (Bash/Git). Native GUI : Hard-coded mouse/keyboard coordinate actions. None (BYOT) : Users must build tools (Search, PDF reader) from scratch. Integration Effort High : Requires solving rigid networking (port mapping) and building a perception bridge to the HTML. Moderate : Standard Docker orchestration; agents speak Terminal natively. Moderate : Authors provide the VM/API, but vision-to-action grounding is a high hurdle. High : The researcher must provide a broad range of tools. Scalability Bottleneck State Reset Latency: VMs require high resources. Benchmark recommends sequential evaluation of all tasks. Disk I/O Churn : Constant building/pulling of unique task images saturates the filesystem. RAM & Snapshots : 20 GB+ memory per agent and heavy disk I/O for state resets. API & Rate Limits : Capped by external search quotas and LLM token costs. To understand why a standard is necessary, consider the engineer bridging a benchmark and a generalist agent, translating raw implementations into agent-compatible interfaces through technical hurdles rarely discussed in papers. Unlike static datasets, agentic benchmarks require live, interactive environments ( 35 ; 24 ) with diverse infrastructure needs spanning web navigation ( 10 ; 43 ; 15 ) , software engineering ( 12 ; 19 ) , operating systems ( 39 ; 5 ) , and mobile devices ( 30 ; 7 ) . Some require shared servers common to all tasks, preventing simple per-task deployment. To make it concrete, Table 1 summarizes the shape and challenges of 4 popular benchmarks. Furthermore, deployment on high-performance computing platforms often runs into port configuration conflicts. Default ports are frequently blocked, requiring custom reconfiguration for every