MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

Wenrui LiuPeking University [email protected], Zixiang LiuColumbia University [email protected], Elsie DaiPeking University [email protected] Wenhan YuPeking University [email protected] Lei YuPeking University [email protected], Tong YangPeking University [email protected] · 2025

Read MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use by Wenrui LiuPeking University [email protected], Zixiang LiuColumbia University [email protected], Elsie DaiPeking University [email protected] Wenhan YuPeking University [email protected] Lei YuPeking University [email protected], Tong YangPeking University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24565.txt.

Introduction

Zixiang Liu and five colleagues benchmark AI agents that use tools through the Model Context Protocol. Agents increasingly call external tools this way. Existing evaluation sets lean on live external services. They also ignore task difficulty. The new benchmark, MCPAgentBench, is built from real-world tool definitions instead.

Authentic tasks pair with simulated tools in a dynamic sandbox. Candidate tool lists include distractors on purpose. Agents must select and discriminate, not just call. Metrics cover task completion and execution efficiency together. Leading language models were tested on the suite. Performance differed sharply on complex, multi-step tool use. Some models handled the chains well; others stumbled. All code is open source. The benchmark aims to make agent tool use measurable without fragile live dependencies.

Catalog record

CREATORWenrui LiuPeking University [email protected], Zixiang LiuColumbia University [email protected], Elsie DaiPeking University [email protected] Wenhan YuPeking University [email protected] Lei YuPeking University [email protected], Tong YangPeking University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24565

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use Wenrui LiuPeking University [email protected], Zixiang LiuColumbia University [email protected], Elsie DaiPeking University [email protected] Wenhan YuPeking University [email protected] Lei YuPeking University [email protected], Tong YangPeking University [email protected] Abstract Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at 9 . † † footnotetext: \dagger Co-first Authors. These authors contributed equally to this work. † † footnotetext: ∗ Corresponding author 1 Introduction Large Language Models (LLMs)( 26 ; 4 ; 11 ; 29 ) achieve breakthrough progress in natural language processing and complex reasoning tasks. To further advance capabilities, the research community shifts focus toward Agent models. The Model Context Protocol (MCP)( 19 ) currently represents a crucial exploration aimed at unifying the interaction modality between Agents and external tools, defining a standard format for tool invocation. Agent utilization of MCP tools becomes a key approach for solving complex, real-world tasks. Consequently, establishing a comprehensive evaluation benchmark that assesses the Planning and Execution capabilities of Agents in invoking MCP tools is essential. However, existing MCP capability assessment benchmarks 14 ; 7 ; 6 suffer from several significant limitations. Firstly, a stability and dependency issue exists, as current benchmarks often rely on real, remote MCP servers, where service stability and availability heavily impact the reproducibility of testing results. Secondly, these benchmarks exhibit insufficient difficulty awareness, performing only simple task categorization and lacking granular observation of the invocation complexity level. Most importantly, models should be able to efficiently complete tasks, while existing frameworks lack metrics for task execution efficiency. To address these challenges, a new evaluation benchmark must incorporate: local MCP server deployment to ensure stability; comprehensive coverage of complex invocation scenarios, including single-step, serial, and parallel calls; and dedicated metrics for task execution efficiency. This paper proposes MCPAgentBench, an evaluation benchmark specifically designed to assess the efficiency of Agent MCP tool invocation, aiming to resolve the aforementioned challenges. This evaluation work provides the following contributions: • Data and Instance Construction: We collect authentic 841 tasks and over 20000 MCP Tools from sources including MCP Marketplace( 15 ), Github( 8 ), and HuggingFace( 17 ). We perform simple local reconstruction of all MCP Tools and, through manual labeling and matching, ultimately construct 180 high-quality task instances. • Automation Framework: An automated evaluation framework implemented based on Autogen achieves dynamic loading of tasks and MCP Tools, which ensures the automation and scalability of the evaluation. • Efficiency Metrics: The Task Finish Score (TFS), Task Efficiency Finish Score (TEFS), Time Efficiency, and Token Efficiency metrics define the comprehensive evaluation of the Agent’s planning correctness, execution timing, and resource consumption. MCPAgentBench incorporates complex tasks spanning daily and professional domains, rigorously distinguishing tasks based on serial and parallel invocation complexity. This benchmark allows for an in-depth evaluation and analysis of mainstream Large Language Models, revealing a common efficiency deficiency in handling complex tool calls, particularly in parallel tool invocations. 2 Related Work Recent advances in agentic large language models (LLMs) have shifted evaluation from pure text generation toward tool-use reasoning, with a growing emphasis on when, how, and why agents invoke tools. The Model Context Protocol (MCP) has emerged as a unified substrate for such evaluation by enforcing schema-consistent communication and execution-grounded validation, enabling reproducible and protocol-aligned assessment of tool-augmented agents. Before MCP, tool-use benchmarks were developed largely within heterogeneous API ecosystems. API-Bank 13 and ToolBench 23 evaluated Plan–Retrieve-Call behaviors but lacked unified protocol abstraction and execution-level guaranties. Earlier paradigms such as ReAct 30 , Auto-GPT 25 , and GAIA 18 explored the interaction between reasoning and acting, though often in synthetic or text-only environments. More recent MCP-based benchmarks represent a clear shift from simulated function calling to execution-verified, protocol-consistent evaluation. MCP-Universe 14 evaluates agents against live MCP servers, MCP-RADAR 7 introduces multi-dimensional evaluation metrics, MCPWorld 28 supports hybrid API–GUI tasks, and MCPToolBench++ 6 scales evaluation to thousands of MCP servers with a fine-grained error taxonomy. While these benchmarks substantially advance the realism and coverage of MCP-based evaluation, they primarily focus on task correctness and protocol compliance. In contrast, MCPAgentBench is proposed as a complementary data set that targets a more fine-grained and decision-centric aspect of tool use: the efficiency with which agents select and invoke MCP tools to complete tasks.In terms of concreteness, MCPAgentBench differs from the existing MCP benchmarks in several key aspects. It employs an Autogen-based sandbox with locally maintained MCP servers to ensure stable and reproducible execution. Tasks are categorized according to the complexity of MCP tool invocation and task attributes, enabling structured analysis across difficulty levels. Moreover, MCPAgentBench uses authentic definitions and parameters of the MCP tool, building simulated MCP servers that strictly follow the standard MCP protocol, while introducing realistic distractors that test the’ robustness of agents in tool selection. Together, these design choices position MCPAgentBench as a complementary benchmark that emphasizes task completion efficiency, task authenticity, and robustness to interference, allowing fine-grained evaluation of agent tool-invocation capabilities beyond correctness alone. 3 MCPAgentBench Framework This section introduces the architecture of MCPAgentBench, the benchmark construction process, the evaluation process, task classification, and evaluation metrics. Figure 1: MCPAgentBench Overview. 3.1 Overview MCPAgentBench employs a sandbox environment built upon the Autogen framework. For each task, the system dynamically loads a corresponding MCP tool list via Autogen’s tool interface to facilitate automated benchmark testing. The overall architecture of the MCPAgentBench framework, illustrated in Figure 1 , comprises three key components: • MCP Tool Collection: A repository of authentic MCP tools, collected and curated from GitHub and various MCP collection websites. MCPAgentBench extracts structured information for each tool, particularly its functional description and parameters. • Task Set: A diverse collection of tasks spanning daily life and professional domains. These tasks, originating from real-world datasets, have undergone meticulous manual review and curation to ensure the uniqueness of each task’s solution. • Automated Evaluation Sandbox: A sandbox environment implemented using the Autogen framework, designed for automated task execution and evaluation. The evaluation process leverages this sandbox environment. For each task T , MCPAgentBench retrieves n corresponding correct tools ( G ) from the main tool library and samples K-n "distractor tools" ( F ) that are functionally unrelated or easily confused. Together, these form a dynamic candidate list L containing K tools (e.g., K=20,30 ), which MCPAgentBench provides to the agent under test at runtime. The agent (driven by the LLM under test) must interpret task T , select the correct tool(s) from the distractor-filled list L

Read the complete text