An Empirical Study of Automating Agent Evaluation
Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong · 2026
Read An Empirical Study of Automating Agent Evaluation by Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11378.txt.
Introduction
Zhou and seventeen colleagues ask whether frontier coding assistants can reliably automate agent evaluation. Judging multi-step tool use and reasoning is costly and expert work. Simply prompting them fails. Without evaluation expertise they succeed on only thirty percent of executions. They also produce bloated evaluations with over twelve metrics per agent. Their EvalAgent encodes evaluation skills into a trace-based pipeline, lifting first-run success from 17 point 5 to 65 percent. The point is that coding skill alone does not make a good judge of agent work.
Catalog record
| CREATOR | Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101308 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of Automating Agent Evaluation Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong Abstract Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent , an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench , a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%. \reportnumber \codeurl https://github.com/awslabs/Agent-EvalKit 1 1 footnotetext: Equal contribution 2 2 footnotetext: Corresponding author: [email protected] 1 Introduction Evaluation is the bottleneck in agent deployment. Autonomous AI agents now operate in high-stakes applications ( 40 ; 35 ) , such as code generation ( 38 ) , scientific discovery ( 45 ) , and financial analysis ( 13 ) . Yet the tools to assess their quality lag far behind the tools to build them. The core difficulty is that agent evaluation differs fundamentally from LLM evaluation. While LLM evaluation focuses on output quality for single-turn or multi-turn conversations ( 46 ; 23 ) , agent evaluation must assess execution traces containing sequences of reasoning steps, tool invocations, error recovery actions, and state transitions that unfold over time ( 17 ; 14 ) . Only evaluating the final output is insufficient because (1) a correct final answer may mask flawed reasoning ( 21 ) and (2) a failed output may not reflect robust error handling. These behavioral nuances are invisible to output-only methods and are precisely what production deployments need to assess. Agent evaluation also demands significant domain expertise and imposes practical overhead. Practitioners must carefully design evaluation criteria tailored to diverse agent behaviors, instrument agent code to capture relevant signals, and manually inspect execution traces to diagnose failures ( 43 ; 26 ) . A common workaround is to embed evaluation logic directly within agent code ( 8 ; 29 ) . However, this tightly couples evaluation with implementation, causing evaluation behavior to diverge between development and production environments. These challenges point to a clear need: automating the agent evaluation process itself. Prior work on LLM-as-a-judge ( 46 ; 23 ) and agentic judge ( 49 ; 17 ; 14 ; 5 ) uses models to directly produce verdicts against pre-defined criteria. But judgment presupposes that criteria already exist. We identify a harder problem: evaluation generation , given an agent’s source code and execution traces, automatically generating the evaluation criteria, executable assessment code, and actionable reports. This requires not just assessing quality, but designing what to assess and how , a challenge that existing approaches leave unaddressed. Can frontier coding assistants ( e.g. , Claude Code ( 2 ) ) automate this process? Our study shows that they cannot do so reliably without intervention, exhibiting three systematic failure modes. First, metric proliferation : without evaluation-specific guidance, they generate 12+ scattered metrics per agent, favoring operational measurements like latency percentiles and token counts over meaningful assessments of task success. Second, code over-engineering : generated evaluation code is 2–3 \times longer than necessary, fragmented across 5–7 files with premature abstractions that undermine maintainability. Third, plan-code drift : they generate evaluation plans with well-reasoned semantic rubrics but implement shallow keyword-counting heuristics instead. These failures reflect a fundamental mismatch between general code generation and the specialized reasoning that evaluation design demands. We present EvalAgent (§ 2 ), a system that transforms agent source code and user requirements into executable evaluation artifacts. EvalAgent encodes evaluation domain expertise as evaluation skills —self-contained packages of procedural instructions, reusable code and templates, and dynamically retrieved API documentation ( 44 ) —that compose into a trace-based pipeline: planning evaluation criteria, generating test scenarios, instrumenting the agent for trace collection, producing executable assessment code, and generating actionable reports. Each pipeline stage loads only the relevant skills on demand, channeling generation toward focused, implementable assessments rather than the scattered metrics and shallow heuristics produced by unconstrained approaches. A key challenge remains: how do we know that EvalAgent produces good evaluations? To assess evaluation quality without ground truth, we introduce a meta-evaluation framework (§ 3 ) using pairwise comparison across five dimensions (requirement fulfillment, metric relevance, code quality, plan quality, and plan-code alignment) alongside AgentEvalBench , a benchmark comprising 20 diverse agents spanning 9 frameworks and 14 application domains, each paired with 2 user requirements and 5 test scenarios. Expert annotators achieve Fleiss’ \kappa=0.923 and prefer EvalAgent in 79.5% (10.5% ties) of comparisons. Across all conditions, EvalAgent achieves an 84% win-tie rate over baselines and 65.0% Eval@1—the rate at which generated evaluations execute successfully and produce substantive results on the first attempt (§ 4.1 ). Through design space ablations (§ 4.2 ) and qualitative analysis (§ 4.3 ), we observed: 1. Evaluation skills stabilize executability as metric count grows (65% Eval@1 at 5 metrics vs. 30–40% for baselines), reliably steer generation away from shallow keyword matching toward meaningful semantic assessment, and keep generated code 2.4–6.6 \times more concise than baselines. 2. Planning without evaluation domain knowledge hurts more than it helps. Directly generating evaluation code outperforms planning-then-implementing (74% win-tie). Unstructured planning expands scope rather than constraining it, producing 4.4 \times longer plans, 6.6 \times more code, and large amounts of unused files. Planning becomes effective when paired with evaluation skills. 3. EvalAgent is particularly effective under underspecified requirements. When users provide only generic instructions like “evaluate this agent”, EvalAgent achieves 95% win-tie rates over baselines by inferring meaningful metrics from observed behavior, compared to 80% when requirements are well-specified and simpler methods can partially compensate. 2 EvalAgent Figure 1 : EvalAgent pipeline. EvalAgent maps agent code and user requirements to an executable evaluation workspace and actionable report through a six-stage process: (1) evaluation planning, (2) test case generation, (3) agent instrumentation, (4) trace collection and processing, (5) evaluation code generation and execution, and (6) reporting with actionable recommendations. Evaluation skills guide each stage. EvalAgent is an agentic system that transforms agent source code and user requirements into executable evaluation code and actionable reports through a six-stage pipeline (Figure 1 ). General-purpose coding agents already possess