The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments
Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen · 2026
Read The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments by Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09032.txt.
Introduction
Logan Ritchie and four co-authors grade top AI models on long tasks in a fake shop world. Old grades test one reply at a time. Real work needs many steps in a row, with tools, checks, and fix ups. A model that chats well may still fail a long job. The team uses a set of work place tasks in a shop sim from Surge.
Each task runs in a live world that reacts to the model's acts. Scores show a rank of agent skills, from plain to hard tasks. Gaps show where top models still stall. A reader who buys or builds agents gets a rank based on work, not chat. It can guide which tasks are safe to hand to an agent now.
Catalog record
| CREATOR | Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-09032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen Abstract The advancement of large language model (LLM) based agents has shifted AI evaluation from single-turn response assessment to multi-step task completion in interactive environments. We present an empirical study evaluating frontier AI models on 150 workplace tasks within a realistic e-commerce RL environment from Surge. Our analysis reveals an empirically-derived hierarchy of agentic capabilities that models must master for real-world deployment: (1) tool use, (2) planning and goal formation, (3) adaptability, (4) groundedness, and (5) common-sense reasoning. Even the best-performing models fail approximately 40% of the tasks, with failures clustering predictably along this hierarchy. Weaker models struggle with fundamental tool use and planning, whereas stronger models primarily fail on tasks requiring contextual inference beyond explicit instructions. We introduce a task-centric design methodology for RL environments that emphasizes diversity and domain expert contributions, provide detailed failure analysis, and discuss implications for agent development. Our findings suggest that while current frontier models can demonstrate coherent multi-step behavior, substantial capability gaps remain before achieving human-level task completion in realistic workplace settings. 1 Introduction The year 2025 has witnessed the transition of AI systems from conversational assistants to more autonomous agents capable of executing actions in real-world environments ( Xi et al. 2023 ; Wang et al. 2024 ) . This evolution raises an important question: how much (economically) useful work can AI agents actually perform? Addressing this question requires evaluation paradigms that go beyond traditional benchmarks. Although assessments such as MMLU ( Hendrycks et al. 2021 ) and HellaSwag ( Zellers et al. 2019 ) measure static knowledge and reasoning, they fail to capture the dynamic, multi-step nature of real-world tasks. Similarly, single-turn instruction-following evaluations ( Zhou et al. 2023 ) cannot measure an agent’s ability to recover from errors, adapt to unexpected situations, or maintain coherent behavior across extended interactions. This gap has motivated interactive evaluation environments where AI agents act through tool use and are assessed on realistic multi-step tasks ( Zhou et al. 2024 ; Xie et al. 2024 ; Mialon et al. 2024 ; Xu et al. 2024 ) . These environments enable evaluations of agentic capabilities that static benchmarks cannot capture. Recent empirical studies of production agents reveal that despite widespread deployment in various industries, reliability remains the primary challenge, with practitioners deliberately constraining agent autonomy to maintain operational stability ( Pan et al. 2025 ) . We present an empirical study of frontier AI models evaluated within an RL environment Corecraft , Inc., simulating an online retailer of high-performance PC components and custom builds. Models assume the role of customer support agents, performing tasks ranging from simple database queries to complex multi-step workflows requiring reasoning about system interactions. Our contributions are: 1. Empirical RL environment evaluation : We evaluated frontier models on 150 workplace tasks, revealing substantial performance gaps even among state-of-the-art models. 2. Hierarchy of agentic capabilities : Through systematic failure analysis, we identify five hierarchical capability levels: tool use, planning, adaptability, groundedness, and common-sense reasoning. Model failures cluster predictably along this hierarchy. 3. Task-centric RL environment design : We describe a principled approach to environment construction that emphasizes task diversity, contributions from real domain experts, and a modular architecture that supports both training and evaluation. 4. Detailed failure taxonomy : We provide concrete examples of failure modes at each hierarchy level, offering useful insights for researchers and model developers. Our findings reveal that even the best-performing models fail approximately 40% of tasks. More importantly, the nature of failures differs systematically: weaker models fail at basic tool use and planning, while stronger models struggle primarily with common-sense reasoning requiring contextual inference beyond explicit instructions. These results contextualize some recent findings that production agents typically execute at most ten steps before requiring human intervention ( Pan et al. 2025 ) , suggesting that current deployment constraints reflect genuine capability limitations. 2 Related Work 2.1 Agent Benchmarks and Evaluation Agent evaluation has evolved substantially from early web interaction benchmarks to comprehensive real-world assessments. MiniWoB++ ( Liu et al. 2018 ) established foundational paradigms for the evaluation of web-based agents with simplified browser tasks. WebArena ( Zhou et al. 2024 ) introduced self-hosted functional websites that incorporate tasks across e-commerce, social forums, and collaborative software. VisualWebArena ( Koh et al. 2024 ) extended this to multimodal settings with tasks requiring visual comprehension. OSWorld ( Xie et al. 2024 ) extended evaluation to full operating system interaction across Ubuntu, Windows, and macOS, with tasks that range from office productivity, system utilities, and creative applications. The benchmark identified GUI grounding and operational knowledge as primary failure modes. The GAIA benchmark ( Mialon et al. 2024 ) took a different approach, designing questions conceptually simple for humans but challenging for AI, requiring real-world interaction, multi-modal reasoning, and web browsing. For software engineering, SWE-bench ( Jimenez et al. 2024 ) evaluates agents on 2,294 real GitHub issues that require bug localization and patch generation. Terminal-Bench ( Terminal-Bench Team 2025 ) evaluates the capabilities of command-line agents in tasks that span scientific workflows, network configuration, cybersecurity, and data analysis, each running in isolated Docker environments. \tau -bench ( Yao et al. 2024a ) uses the pass@k reliability metric that measures the success rate over k independent trials rather than best-of- k attempts, with the results surfacing considerable reliability concerns. The follow-up \tau^{2} -bench ( Barres et al. 2025 ) extends the evaluation to dual-control environments where both the agent and the user can invoke tools. Recent surveys ( Yehudai et al. 2025 ; Mohammadi et al. 2025 ) have proposed taxonomies organizing agent evaluation by objectives (behavior, capabilities, reliability, safety) and process (interaction modes, benchmarks, metrics). Our work complements these taxonomies by providing an empirically-derived capability hierarchy that explains why agents fail, not just whether they succeed. 2.2 Production Agent Deployment Although academic benchmarks provide controlled evaluation settings, understanding real-world agent deployment is equally important. Pan et al. 2025 present the first large-scale systematic study of AI agents in production, surveying 306 practitioners and conducting 20 in-depth case studies in 26 domains. Their findings reveal that production agents are typically built using simple controllable approaches: 68% execute at most 10 steps before requiring human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. In particular, reliability remains the top development challenge, and practitioners deliberately constrain agent autonomy to maintain operational stability. These production patterns provide an important context for our evaluation findings. The capability gaps we identify help explain why practitioners adopt bounded autonomy and extensive human oversight: current models have not yet achieved the reliability required for extended autonomous operation. 3 Environment Design 3.1 Design Philosophy Effective evaluation environments for LLM agents should satisfy the following requirements: (1) a coherent world model that defines the setting, organizational relationships, and domain constraints; (2) realistic entities with authentic attributes, relationships, and behaviors; and (3) a functional tool system that allows agents to perceive and act within the environment. We adopt a task-centric design philosophy : tasks provide a useful training and evaluation signal, while entities and tools exist to support tasks. This principle