OS-Map : How Far Can Computer-Using Agents Go in Breadth and Depth?
Xuetian Chen Yinghao Chen Xinfeng Yuan Zhuo Peng Lu Chen, Yuekeng Li Zhoujia Zhang Yingqian Huang Leyan Huang Jiaqing Liang, Tianbao Xie Zhiyong Wu Qiushi Sun Biqing Qi Bowen Zhou · 2025
Read OS-Map : How Far Can Computer-Using Agents Go in Breadth and Depth? by Xuetian Chen Yinghao Chen Xinfeng Yuan Zhuo Peng Lu Chen, Yuekeng Li Zhoujia Zhang Yingqian Huang Leyan Huang Jiaqing Liang, Tianbao Xie Zhiyong Wu Qiushi Sun Biqing Qi Bowen Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19132.txt.
Introduction
Xuetian Chen and co-authors build a benchmark that measures how far computer-using agents can really go. AI agents that click, type, and navigate real software promise big productivity gains. Yet existing tests ignore how tasks differ in kind and how those differences match what users actually need.
OS-Map organizes four hundred sixteen realistic tasks across fifteen applications along two axes. One is a five-level ladder of automation depth. The other is a scope of generalization drawn from a hierarchy of real user demands. The design exposes exactly which capabilities an agent has and lacks, rather than one blended score. For developers, it turns 'the agent works' into a map of where it works, and why.
Catalog record
| CREATOR | Xuetian Chen Yinghao Chen Xinfeng Yuan Zhuo Peng Lu Chen, Yuekeng Li Zhoujia Zhang Yingqian Huang Leyan Huang Jiaqing Liang, Tianbao Xie Zhiyong Wu Qiushi Sun Biqing Qi Bowen Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19132
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OS-Map : How Far Can Computer-Using Agents Go in Breadth and Depth? Xuetian Chen Yinghao Chen Xinfeng Yuan Zhuo Peng Lu Chen, Yuekeng Li Zhoujia Zhang Yingqian Huang Leyan Huang Jiaqing Liang, Tianbao Xie Zhiyong Wu Qiushi Sun Biqing Qi Bowen Zhou Abstract Computer-using agents have shown strong potential to boost human productivity and enable new application forms across platforms. While recent advances have led to usable applications, existing benchmarks fail to account for the internal task heterogeneity and the corresponding agent capabilities, as well as their alignment with actual user demands—hindering both targeted capability development and the reliable transition of research progress into practical deployment. To bridge the gap, we present OS-Map , a benchmark for daily computer-using automation that organizes its 416 realistic tasks across 15 applications along two key dimensions: a five-level taxonomy of automation and a generalization scope derived from a real-world user demand hierarchy. To enable fine-grained analysis of required capabilities and alignment with real-world scenarios, OS-Map evaluates agents along two dimensions: automation level across a five-level taxonomy, and generalization scope across a demand hierarchy. This design captures varying levels of required agent autonomy and generalization, forming a performance–generalization evaluation matrix for structured and comprehensive assessment. Experiments show that even State-of-the-Art agents with VLM backbones struggle with higher-level tasks involving perception, reasoning, and coordination—highlighting the need for a deeper understanding of current strengths and limitations to drive the future progress in computer-using agents research and deployment. All code, environments, baselines, and data are publicly available at https://github.com/OS-Copilot/OS-Map . 1 Introduction Computer-using agents, which can understand user intent and autonomously perform operations across digital environments, is driving the next transformation in human-computer interaction [ 1 , 2 ] . Powered by the extensive world knowledge, interaction capability, and tool-use abilities of Large Language Models (LLMs) and Vision Language Models (VLMs), computer-using agents such as Operator [ 3 ] , Claude 3.5 [ 4 ] , UFO 2 [ 5 ] , and UI-TARS [ 6 ] can understand natural language instructions and interact directly with various applications in a human-like manner. Once a fixture of science fiction—like J.A.R.V.I.S. in Iron Man , seamlessly managing schedules, editing documents, shopping across websites, and automating routine computer tasks—such digital personal assistants are now becoming a tangible reality [ 7 ] . This transformation frees humans to focus on creative work, significantly boosting productivity and enabling new applications. As research on computer-using agents continues to advance, an increasing number of models with strong functionalities [ 6 , 8 , 9 , 10 , 11 ] and agent systems [ 12 , 13 , 14 , 5 , 15 ] are being proposed. Despite the rapid emergence of new methods, the open-ended semantics and diverse capability demands of computer-using tasks still hinder actual deployment [ 1 ] . To effectively bridge this research-to-practice gap, it is crucial to develop a principled benchmark that allows the community to quantify agent capabilities, identify specific failure points, and guide future development. However, existing benchmarks fall short of this goal. While spanning various platforms and scenarios [ 16 , 17 , 18 , 19 ] , they treat tasks as flat collections, without decomposing task heterogeneity and required capabilities [ 20 , 21 , 22 , 19 ] , making it difficult to perform fine-grained evaluation and differentiation. Moreover, task collections are typically organized around applications [ 23 , 21 ] rather than aligned with the actual distribution of daily computer use, limiting the relevance of benchmark performance to real-world utility [ 1 ] . Figure 1: OS-Map qualitative evaluation matrix, summarizing how different types of agents perform across two dimensions. General-purpose models show strong generalization, while scenario experts excel at specific tasks. Mainstream computer-using agents aim to balance both, yet still face major challenges. Agent positioning is based on reported performance, as detailed in Appendix A . To bridge these gaps, we present the OS-Map benchmark that is grounded in dynamic desktop environments and structured along two key dimensions: automation levels and generalization scopes . First, we propose a five-level capability taxonomy based on degrees of autonomy, encompassing a wide range of computer-using tasks—from atomic execution and simple planning to disturbance adaptation, complex orchestration, and proactive behaviors. Second, we derive a real-world user demand hierarchy on daily computer-using scenarios and select representative tasks to ensure both high coverage and alignment with practical demands. Furthermore, we combine the two dimensions into a unified evaluation matrix (Figure 1 ), which highlights how general-purpose models, scenario experts, and mainstream computer-using agents differ in capability trade-offs between automation and generalization. The upper-right corner marks a high-value region—representing impactful yet unachieved applications—where no current agent demonstrates sufficient capability. Across 416 tasks spanning 15 applications in OS-Map , even the strongest existing computer-using agents achieve only an 11.4% overall success rate, with near-zero performance on higher-level tasks—falling far short of human performance. These findings underscore the importance of a principled evaluation framework. By offering both qualitative and quantitative insights into where and to what extent computer-using agents can assist humans, our framework supports comprehensive evaluation and provides a clear roadmap for future progress. 2 Environment OS-Map adopts and extends the OSWorld [ 21 ] infrastructure, which centers around a virtual machine (VM) and a host-side controller (VMC). This dynamic and executable environment offers fine-grained control, consistent reproducibility, flexible extensibility, and secure isolation, forming an ideal sandbox for evaluating computer-using agents in real-world scenarios. 2.1 Task Definition In general, computing-using automation tasks are roughly modeled as partially observable Markov decision processes \left(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R}\right) . At timestep t , the agent resides in the environment state s_{t}\in\mathcal{S} , but only receives a partial observation o_{t}\in\mathcal{O} ( e.g. , the current screenshot). Based on o_{t} , the agent emits an action a_{t}\in\mathcal{A} ( e.g. , a structured text click(350,600) ). The environment transitions to the next state s_{t+1}=\mathcal{T}(s_{t},a_{t}) via the transition function \mathcal{T} , which is governed by the underlying software and OS logic, revealing a new observation o_{t+1} . This process continues iteratively until the agent actively issues a terminal action ( i.e. , DONE or FAIL ), or passively exceeds a predefined step limit. After termination, the system determines whether the task is successfully completed and provides a final outcome reward r\in\mathcal{R}=\{0,1\} , without any intermediate process rewards. Figure 2: OS-Map is built on an executable desktop environment designed for daily computer tasks, integrating a suite of applications and tools. It provides the infrastructure for reliable evaluation by handling task initialization and success verification. Agents interact autonomously via GUI operations, guided by instructions and screenshot perception. 2.2 Environment Structure Task lifetime. Each task is specified by a JSON file defining initialization, instruction, and evaluation protocols. As shown in Figure 2 , evaluation begins by restoring a designated VM snapshot and running lightweight setup routines. The agent then enters the interaction loop, receiving observations from and sending actions to the VM via the VMC. This loop continues until the agent terminates the episode, either actively or passively. The evaluator then compares the VM state to reference criteria and returns a binary reward. See Appendix B.1 for details. Initialization and evaluation configuration. Task setup in OS-Map combines VM snapshots with modular configuration functions, supporting