MCPWorld : A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents
Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, Mengwei Xu · 2025
Read MCPWorld : A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents by Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, Mengwei Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07672.txt.
Introduction
Yunhe Yan and co-authors built a testbed in 2025 for computer use agents, the AI helpers that operate software the way a person would. Older test sets graded only screen clicking agents, which break when layouts change, and ignored the direct function calls many apps now expose.
Their testbed uses open source apps that can be rebuilt with those function hooks added. That lets the system check task success by watching the app's inner state, not the screen. It ships with 201 hand checked tasks of varied difficulty, packaged to run anywhere with graphics support. A first trial with a leading agent framework finished about three quarters of the tasks. The authors offer it as shared ground for comparing designs.
Catalog record
| CREATOR | Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, Mengwei Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MCPWorld : A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, Mengwei Xu Abstract (M)LLM-powered computer use agents (CUA) are emerging as a transformative technique to automate human-computer interaction. However, existing CUA benchmarks predominantly target GUI agents, whose evaluation methods are susceptible to UI changes and ignore function interactions exposed by application APIs, e.g., Model Context Protocol (MCP). To this end, we propose MCPWorld , the first automatic CUA testbed for API, GUI, and API-GUI hybrid agents. A key principle of MCPWorld is the use of “white-box apps”, i.e., those with source code availability and can be revised/re-compiled as needed (e.g., adding MCP support), with two notable advantages: (1) It greatly broadens the design space of CUA, such as what and how the app features to be exposed/extracted as CUA-callable APIs. (2) It allows MCPWorld to programmatically verify task completion by directly monitoring application behavior through techniques like dynamic code instrumentation, offering robust, accurate CUA evaluation decoupled from specific agent implementations or UI states. Currently, MCPWorld includes 201 well curated and annotated user tasks, covering diversified use cases and difficulty levels. MCPWorld is also fully containerized with GPU acceleration support for flexible adoption on different OS/hardware environments. Our preliminary experiments, using a representative LLM-powered CUA framework, achieve 75.12% task completion accuracy, simultaneously providing initial evidence on the practical effectiveness of agent automation leveraging MCP. Overall, we anticipate MCPWorld to facilitate and standardize the benchmarking of next-generation computer use agents that can leverage rich external tools. Our code and dataset are publicly available at https://github.com/SAAgent/MCPWorld . 1 Introduction Computer Use Agents (CUAs) ( computeruse ) powered by (multimodal) large language models (LLMs) are rapidly emerging as a transformative technique that augment human productivity. CUAs automate routine tasks, such as document processing, message sending, and application configuration, by understanding natural language instructions and executing them through interaction with computers. In general, interactions between CUAs and the underlying environment fall into two categories: GUI and API. A CUA leverages one of them, or a hybrid of both, as observation and action spaces, to accomplish user tasks. With the standardization of CUA-to-API interface, e.g., model context protocol (MCP) ( mcp_intro ) , we anticipate API-powered CUAs to play an increasingly important role in the future. However, we find that existing benchmarking tools and testbeds mostly target GUI agents only ( osworld ; windowsagentarena ; androidworld ) . Unlike CUAs performing solely GUI operations, constructing and benchmarking API-powered CUAs needs to (i) provide intrusive visibility into app internals and permission to define APIs; (ii) devise a GUI-independent evaluator that can accurately judge task completion without looking at the device screen. Currently, only a handful of testbeds provide support for the API CUAs. Several prior studies ( api_bank ; toolbench ; toolalpaca ; toolqa ; taskmatrix ) investigated the capability of API CUAs. But only one recent work, AgentStudio ( agentstudio ) supports comparison between two interfaces, but only through simple queries of external output files with no MCP support. To fill the research gap, we present MCPWorld , the first MCP-enabled benchmarking toolkit and testbed for API, GUI, and hybrid Desktop CUAs. A key principle of MCPWorld is the use of “white-box apps”, i.e., those with source code availability and can be revised/re-compiled as needed (e.g., adding MCP support). Such a design choice not only greatly broadens the design space of CUA, such as what and how the app features to be exposed/extracted as CUA-callable APIs, but also allows MCPWorld to programmatically verify task completion by directly monitoring application behavior through techniques like dynamic code instrumentation, offering robust, accurate CUA evaluation decoupled from specific agent implementations or UI states. Table 1 compares MCPWorld to prior CUA testbeds, highlighting its uniqueness in the app selection and evaluation methods. We conducted considerable effort in building MCPWorld . At its core, MCPWorld provides a fully containerized desktop environment that ensures reproducible testing conditions, with properly desktop applications setup. Atop the environment with applications, MCPWorld integrates two sets of unified tools for GUI and MCP interaction. GUI tool abstracts the functionality of accessing varying GUI representations (e.g., pixel-level screenshots, textual view hierarchies), while leaving the flexibility for customizing app-specific observation spaces. The MCP tool provide a general framework for direct function-call style interaction with applications; for our white-box applications, we integrated existing MCP servers to expose a rich set of internal operations as standardized APIs for agent use. For MCPWorld evaluator, we propose and implement three different task verification methods: dynamic binary instrumentation, targeted code injection, and API-driven state querying, all carefully tailored to each application’s design. These verification methods synchronize directly with application internal changes, operate independently of the CUA’s workflow, and provide immediate feedback on CUA-environment interactions, thus enabling precise, real-time evaluation. MCPWorld benchmark suite encompasses 201 carefully curated tasks with varying complexity levels across 10 widely-used desktop applications (e.g., VS Code and OBS Studio). Tasks are derived and annotated based on realistic user scenarios by 12 authors with computer science backgrounds. For each task, we performed in-depth source code analysis, developed custom verification hooks into application internals, and designed application state snapshots to ensure consistent evaluation. Tasks range from simple configurations to complex multi-step workflows, with difficulty categorized by human interaction steps and balanced coverage across usage categories. Each task features detailed natural language instructions and fine-grained progress tracking signals through annotated key milestones, enabling precise evaluation of agent performance. MCPWorld is designed for scalability and ease of use. We provide standardized workflows and helper tools that simplify the addition of new applications, tasks and toolsets. For agents already supporting the MCP, integration with MCPWorld becomes exceptionally straightforward, allowing researchers to rapidly benchmark their capabilities. We built a CUA on top of Claude Computer Use computeruse to test its performance on MCPWorld . Our preliminary results show that Claude Computer Use agent with Claude 3.7 Sonnet achieves 75.12% task completion accuracy. We further provide initial evidence on the practical effectiveness of agent automation leveraging MCP. Overall, we anticipate MCPWorld could facilitate and standardize the benchmarking of next-generation CUAs that can leverage rich external tools. Table 1 : Comparing different benchmarks and datasets for computer use agent evaluation. The meanings of symbols: : all apps used are open-sourced; : all apps used are closed-sourced; : not all apps used are open-sourced; * : tasks are provided in the form of templates and can be expanded into different tasks according to different task parameters. Compared to our novel evaluation method (App internals hooking), we summarize the previous evaluation methods as “external state matching”, which can be mainly divided into three specific categories: step-wise action matching, external UI matching and output file matching. Name App/Website/Environment # tasks Evaluation Method Platform Supported Agent Action Space Number Open Source MCP Enabled AndroidWorld ( androidworld ) 20 ✗ 116 * All three external state matching Android GUI AndroidArena ( androidarena ) 13 ✗ 221 Step-wise action matching Android GUI LlamaTouch ( llamatouch ) 57 ✗ 496 Output file & external UI matching Android GUI A3 ( a3 ) 20 ✗ 201