MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang Xiangyu Yue, Weijie Su, Xizhou · 2025
Read MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents by Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang Xiangyu Yue, Weijie Su, Xizhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19478.txt.
Introduction
Xuehui Wang and colleagues released a benchmark in 2025 that grades computer-using agents the way a patient examiner would, level by level. Their framework, MMBench-GUI, spans Windows, macOS, Linux, iOS, Android, and the web. It has four levels: understanding what is on screen, locating the right element, automating a task, and collaborating across tasks. Each level isolates a skill that real automation needs.
The team also proposes an efficiency-quality area metric, which scores how well an agent balances speed against success in live automation. Their evaluations identified accurate visual grounding, pinpointing the right screen element, as a critical bottleneck for current agents. As software agents move from demos to daily tools, hierarchical benchmarks like this show developers exactly which floor of the building is unfinished.
Catalog record
| CREATOR | Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang Xiangyu Yue, Weijie Su, Xizhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19478
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang Xiangyu Yue, Weijie Su, Xizhou Zhu, Wei Shen, Jifeng Dai, Wenhai Wang Abstract We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels—GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration—covering essential skills for GUI agents. In addition, we propose a novel Efficiency–Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI . 1 Introduction With the rapid advancement of Vision-Language Models (VLMs) ( Wang et al. 2024 ; Chen et al. 2024b ; Bai et al. 2025 ; Zhu et al. 2025 ; Team et al. 2025 ; Xiaomi 2025 ) , the capability of agents to perform complex interactions within Graphical User Interfaces (GUIs) has significantly improved ( Wu et al. 2024a ; Cheng et al. 2024 ; Hong et al. 2024 ; Zheng et al. 2024 ; Gou et al. 2024 ) . These GUI agents have shown great potential in automating complex, repetitive tasks across various domains, substantially enhancing productivity ( Xu et al. 2024b ; Wu et al. 2024b ; Lin et al. 2024 ; Qin et al. 2025 ; Yang et al. 2024 ) . However, widely adopted evaluation benchmarks ( Zhou et al. 2023 ; Cheng et al. 2024 ; Xie et al. 2024 ; Li et al. 2024 ; Chang et al. 2024 ; Rawles et al. 2024 ; Li et al. 2025 ; Nayak et al. 2025 ; Sun et al. 2025 ; Xie et al. 2025 ) currently face several critical limitations that hinder the further development of GUI agents: (1) existing benchmarks predominantly evaluate isolated capabilities and do not comprehensively analyze the agents’ overall capabilities and the relationships between multiple capabilities ( Deng et al. 2023a ; Cheng et al. 2024 ; Xie et al. 2025 ; Li et al. 2025 ) ; (2) current evaluation metrics primarily emphasize task accuracy and success rate, overlooking operational efficiency ( Zhou et al. 2023 ; Xu et al. 2024a ; Xie et al. 2024 ; Bonatti et al. 2024 ) ; (3) insufficient coverage of evaluation scenarios fails to fully represent widely used GUI systems in real-world applications ( He et al. 2024 ; Xie et al. 2024 ; Rawles et al. 2024 ; Sun et al. 2025 ) . Figure 1: MMBench-GUI : a hierarchical benchmark spanning four levels of increasing difficulty, covering over 8,000 tasks across six commonly used platforms. From L1 to L4, task complexity increases progressively, placing growing demands on the agent’s generalization and reasoning abilities. Based on this benchmark, we visualize the performance of various models in the right figure, clearly illustrating their respective strengths as well as areas with substantial room for improvement. To address these issues, we propose MMBench-GUI, a hierarchical, multi-platform benchmark framework designed for the systematic evaluation of GUI agents. As shown in Figure 1 , this framework comprises four progressive evaluation levels: (1) GUI Content Understanding, (2) GUI Element Grounding, (3) GUI Task Automation, and (4) GUI Task Collaboration. Each level addresses critical capabilities from basic interface understanding to complex cross-application task execution, ensuring comprehensive and systematic evaluation. Additionally, we introduce the Efficiency-Quality-Aware (EQA) metric, which evaluates both task accuracy and operational efficiency, encouraging agents to complete tasks with minimal interaction steps. Furthermore, to ensure practical relevance, we have constructed a multi-platform dataset covering Windows, macOS, Linux, iOS, Android, and Web platforms, effectively reflecting diverse real-world scenarios. Leveraging extensive evaluations and analysis with MMBench-GUI, we identify key limitations and reveal the current state of GUI agents. Our findings indicate that: (1) although general language models excel at high-level planning and reasoning tasks, they significantly lag in precise visual interaction capabilities. Precise visual grounding is identified as a core determinant of task success, underscoring the critical need for enhanced localization accuracy; (2) efficiency now eclipses raw success rate as the next hurdle. Our EQA metric exposes the considerable surplus steps incurred by contemporary agents during task execution–redundancies that stem from localization inaccuracies, incomplete action spaces, and short-sighted or inadequate planning; (3) agent performance notably degrades when faced with complex, ambiguous, and cross-application tasks, exposing weaknesses in memory management, state tracking, and adaptive reasoning mechanisms. Addressing these shortcomings is crucial for future advancements in GUI agents. In summary, our primary contributions are as follows: • We propose a cross-platform, hierarchical benchmark designed to comprehensively evaluate GUI agents across multiple task types and difficulty levels. Inspired by a human-centered perspective, the benchmark is structured in a progressive manner, covering four essential capabilities. For static tasks (L1 and L2), we introduce fine-grained difficulty stratification to enable stepwise assessment. For dynamic tasks (L3 and L4), we provide both cleaned data splits and novel task constructions to better reflect real-world variability. • We develop the first evaluation benchmark that spans all widely used operating systems, including Windows, Linux, macOS, Android, iOS, and the Web. To the best of our knowledge, this is the first work to enable consistent multi-platform evaluation of GUI agents under a unified protocol. This broad coverage allows for more realistic and application-aligned performance assessment. Notably, our benchmark is also the first to include online task scenarios for macOS, filling a long-standing gap in GUI agent evaluation. • We introduce a novel metric, Efficiency-Quality-Aware (EQA), to jointly assess both the success and efficiency of agent behavior in online tasks. While most prior works focus solely on success rate (SR), EQA additionally considers when the task is completed within the step budget, providing a measure of action redundancy. This metric offers deeper insights into agent behavior and promotes the development of agents that are not only capable but also efficient—an often overlooked dimension in prior benchmarks. 2 Related works GUI Agents. GUI agents have attracted growing interest, driven by advances like Anthropic’s Computer-Use Agent 1 1 1 https://www.anthropic.com/news/3-5-models-and-computer-use and OpenAI’s Operator 2 2 2 https://openai.com/index/computer-using-agent . Currently, GUI Agents mainly fall into two paradigms: a) Modular agent schemes ( Cheng et al. 2024 ; Gou et al. 2024 ; Yang et al. 2024 ; Zhang et al. 2025 ; Wu et al. 2025 ; Xie et al. 2025 ; Wang et al. 2025 ) , which