Automatically Benchmarking LLM Code Agents through Agent-driven Annotation and Evaluation
Lingyue Fu 1 , Bolun Zhang 1 , Hao Guan 1 , Yaoming Zhu 3 , Lin Qiu 2 , Weiwen Liu 1 , Xuezhi Cao 2 , Xunliang Cai 2 , Weinan Zhang 1 , Yong Yu 1,∗ · 2025
Read Automatically Benchmarking LLM Code Agents through Agent-driven Annotation and Evaluation by Lingyue Fu 1 , Bolun Zhang 1 , Hao Guan 1 , Yaoming Zhu 3 , Lin Qiu 2 , Weiwen Liu 1 , Xuezhi Cao 2 , Xunliang Cai 2 , Weinan Zhang 1 , Yong Yu 1,∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24358.txt.
Introduction
Lingyue Fu and nine co-authors propose an agent driven pipeline to build benchmarks for code agents automatically. Code agents now develop software at project level with language models, but evaluating them is hard. Good project level test sets need deep expertise, making annotation costly and narrow, while agent judges using plain in context learning often misjudge quality against human standards.
Their pipeline uses agents themselves to annotate and evaluate, cutting the cost of benchmark construction and widening coverage. It also aims to align automated judgments more closely with human standards than generic judges achieve. The work addresses a growing bottleneck in measuring real software engineering ability.
Catalog record
| CREATOR | Lingyue Fu 1 , Bolun Zhang 1 , Hao Guan 1 , Yaoming Zhu 3 , Lin Qiu 2 , Weiwen Liu 1 , Xuezhi Cao 2 , Xunliang Cai 2 , Weinan Zhang 1 , Yong Yu 1,∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automatically Benchmarking LLM Code Agents through Agent-driven Annotation and Evaluation Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus Thanks: * corresponding author. CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) Lingyue Fu 1 , Bolun Zhang 1 , Hao Guan 1 , Yaoming Zhu 3 , Lin Qiu 2 , Weiwen Liu 1 , Xuezhi Cao 2 , Xunliang Cai 2 , Weinan Zhang 1 , Yong Yu 1,∗ Abstract . Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs) and widely adopted tools. However, existing benchmarks for code agent evaluation face two major limitations: high annotation cost and expertise requirements, and rigid evaluation metrics that rely primarily on unit tests. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse and challenging project-level tasks. Based on this approach, we introduce PRDBench, a novel benchmark comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Document (PRD) requirements, comprehensive evaluation criteria, and reference implementations. PRDBench features rich data sources, high task complexity, and flexible metrics. We further employ an Agent-as-a-Judge paradigm to score agent outputs, enabling the evaluation of various test types beyond unit tests. Extensive experiments on PRDBench demonstrate its effectiveness in assessing the capabilities of both code agents and evaluation agents, providing a scalable and robust framework for annotation and evaluation. C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction In recent years, code agents have made significant progress, capable of solving increasingly complex programming tasks. From initially focusing on single-file code generation to now advancing towards complete project-level software development, Code Agents are demonstrating powerful software development capabilities. LLM-based Code Agents such as CursorAgent ( Inc. 2025 ) , Claude Code ( Anthropic 2025a ) and Gemini CLI ( Mullen and Salva 2025 ) have been widely adopted in real-world development scenarios, accelerating the proliferation of automated programming tools. As code agents continue to advance, a variety of benchmarks have emerged to evaluate their capabilities. Several works ( Chan et al. 2024 ; Huang et al. 2024 ; Daghighfarsoodeh et al. 2025 ) focus on assessing LLMs and code agents in data science tasks, including subtasks such as data processing and machine learning training. Other benchmarks ( Jimenez et al. 2023 ; Fu et al. 2025 ; Vergopoulos et al. 2025 ) select complex code repositories from GitHub pull requests and transform them, often with minimal human intervention, into testable scenarios. In addition, PaperBench ( Starace et al. 2025 ) evaluates the ability of agents to reproduce paper-level code implementations, providing a more rigorous assessment of their research reproducibility. Despite these advancements, current benchmarks face two major limitations. First, the creation of high-quality evaluation datasets requires substantial human effort and expertise . Generating reliable unit tests or grading tasks often depends on expert annotators, especially when migrating real-world scenarios such as Kaggle competitions or GitHub pull requests. These tasks demand domain-level verification and annotation to produce usable test cases. As code agents advance rapidly, the requirements for annotators’ expertise and time investment continue to increase. For example, PaperBench ( Starace et al. 2025 ) recruited ICML authors as annotators, with each annotation task demanding several days of work from PhD-level experts. This process not only incurs high annotation costs, but also restricts the diversity of evaluation data, as recruiting experts from various domains is challenging and often leads to datasets being sourced from a single domain. Second, the rigidity of the evaluation limits the broad applicability of current benchmarks . Existing benchmarks ( Jimenez et al. 2023 ) primarily rely on unit test pass rates to assess code agent performance. While unit tests are effective for verifying specific functions or components, they are limited in scope and cannot fully cover the diverse testing needs encountered in project-level software development. In practical scenarios, quality assurance involves not only unit testing, but also integration testing, end-to-end testing, performance testing, and security testing, which are all essential for robust software delivery. Relying solely on unit tests is not only insufficient, but also overly restrictive, as it enforces strict requirements on project interfaces and implementation details. Many real-world tasks require broader validation methods. To address these gaps, it is necessary to introduce additional evaluation strategies such as shell interaction and file comparison, enabling more comprehensive and flexible assessment of code agents in complex development environments. Benchmark Area # Project Annotator # Metrics Judger Claude Code Score SWE-Bench ( Jimenez et al. 2023 ) Pull Requests 12 Web Crawler 2,294 Unit Test 70.3% ∗ MLEBench ( Chan et al. 2024 ) AI Competition 75 Human 75 Test Set 51.1% DevAI ( Zhuge et al. 2024 ) AI Developement 55 Human 365 Agent 73.0% † PaperBench ( Starace et al. 2025 ) AI Research 20 Human (PhD.) 8,316 Human/LLM 21.0% PRDBench Engineering Development 50 Agent & Human 1,262 Agent 45.5% Table 1. Comparison between PRDBench and project-level code agent benchmarks. We employ the performance of code agents with Claude as a benchmark indicator for assessing the task difficulty of our proposed evaluation suite. The results from external report ( Anthropic 2025a ) is denoted by * , from our implementation is denoted by \dagger , otherwise we use the best results from the original paper with author specified Claude-driven agents. To address the two aforementioned challenges, we propose an agent-driven construction approach that enables the creation of project-level benchmarks with flexible metrics at low human cost. Specifically, a state-of-the-art code agent is used to generate both the project scaffolding and a Product Requirement Document (PRD) along with an executable criteria scheme. Human annotators are only required to verify whether the criteria scheme is compatible with the scaffolding interfaces and whether the expected outputs are reasonable, without the need to manually create detailed evaluation standards or reference solutions. This significantly reduces annotation complexity: for PRDBench, annotators with undergraduate-level knowledge in software engineering related fields are able to complete the annotation, with an average of only eight hours needed to finish the scaffolding and metrics for each project, greatly improving annotation efficiency. Based on this approach, we build a Product Requirement Document (PRD)-centered benchmark, named PRDBench 1 1 1 Code of PRDBench Evaluation is available in https://github.com/AGI-Eval-Official/PRDBench. PRDBench data will be available soon. . PRDBench consists of 50 real-world coding tasks, each defined by a structured PRD, an well-specified and verifiable criteria scheme, and a standard solution code repository. The coding tasks are sourced from real-world requirements, academic projects, and thesis work, spanning 20 common subdomains. For each task, the PRD provides a criteria scheme that facilitates comprehensive human-like quality assurance (QA) checks. We present a comparison between PRDBench and previous code agent benchmarks in Table 1 , where PRDBench offers a comprehensive, multifaceted, project-level benchmark for code agents. During evaluation, we employ the Agent-as-a-Judge paradigm to score code according to the criteria scheme, enabling the evaluation of a wide range of test types beyond unit tests. In summary, our contributions include: • We design an agent-driven data production pipeline , where