Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
Miaosen Zhang Xiaohan Zhao Zhihong Tan Huoshen Zhou Yijia Fan Yifan Yang, Kai Qiu Bei Liu Justin Wagle Chenzhong Yin Mingxi Chen Ji Li Qi Dai, Chong Luo Xu Yang Xin Geng Baining Guo · 2026
Read Covering Human Action Space for Computer Use: Data Synthesis and Benchmark by Miaosen Zhang Xiaohan Zhao Zhihong Tan Huoshen Zhou Yijia Fan Yifan Yang, Kai Qiu Bei Liu Justin Wagle Chenzhong Yin Mingxi Chen Ji Li Qi Dai, Chong Luo Xu Yang Xin Geng Baining Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12501.txt.
Introduction
Zhang and sixteen colleagues tackle the long tail of computer-use failures, where rare, complex interactions cause a large share of errors. They argue the root cause is scarce training data for such interactions. Their CUActSpot benchmark tests complex actions across five modalities: interface, text, table, canvas and natural image, going beyond click-only tests centred on widgets. A renderer-based pipeline then synthesises scenes for each modality, recording screenshots and element positions, and a language model proposes tasks. The design aims to broaden coverage of hard interactions.
Catalog record
| CREATOR | Miaosen Zhang Xiaohan Zhao Zhihong Tan Huoshen Zhou Yijia Fan Yifan Yang, Kai Qiu Bei Liu Justin Wagle Chenzhong Yin Mingxi Chen Ji Li Qi Dai, Chong Luo Xu Yang Xin Geng Baining Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76244 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12501
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Covering Human Action Space for Computer Use: Data Synthesis and Benchmark Miaosen Zhang Xiaohan Zhao Zhihong Tan Huoshen Zhou Yijia Fan Yifan Yang, Kai Qiu Bei Liu Justin Wagle Chenzhong Yin Mingxi Chen Ji Li Qi Dai, Chong Luo Xu Yang Xin Geng Baining Guo Abstract Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models’ capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git . Figure 1: Overview. Prior GUI grounding research (lower-left panel of the inset) is dominated by click actions on standard GUI widgets. Computer-use agents, however, need to operate across a broader action space including editing tables, manipulating text, drawing on canvases, annotating images, and execute richer actions including dragging and freehand drawing. We study this gap through four contributions: benchmark, data-synthesis pipeline, training models, and empirical studies. 1 Introduction Computer-Using Agent (CUA) [ 1 , 2 ] is a key direction for liberating human labor in digital work and enhancing productivity. CLI-based and GUI-based paradigms constitute two major interaction modes for CUAs. Compared with CLI-based CUAs, GUI-based CUAs inherently offer near-zero-cost cross-platform generalization, more user-friendly human–agent collaboration, and a higher theoretical ceiling: in principle, any computer task that humans can accomplish could also be completed by GUI-based CUAs. However, owing to their efficiency and LLM-friendly interaction format, CLI-based CUAs [ 3 , 4 , 5 ] have already demonstrated practical applicability faster than GUI-based. Ideally, future CUAs will evolve into hybrid systems that combine the efficiency of CLI-based interaction with the flexibility and freedom of GUI-based operation. This paper primarily investigates the practical bottlenecks that hinder the deployment of GUI-based CUAs in real-world applications. Figure 2: Upper: Failure studies of GPT-5.4 computer use. \dagger : System errors refer to failures arising from the stochastic instability or limited robustness of certain VMs and interaction tools (e.g., PyAutoGUI). Lower: An E2E bad example of GPT-5.4 making a PPT introducing itself. We begin with a user study of GPT-5.4’s [ 6 ] computer-use capability on the Azure OpenAI platform. We collected nearly 200 tasks spanning three scenarios: work, web usage [ 7 ] , and gaming [ 8 , 9 ] , and executed them in a Windows VM, analyzing all failure cases that except system errors. As summarized in the upper part of Figure 2 , we find that Action Grounding [ 10 , 11 , 12 , 13 ] is the most important source of error in the work setting, which is also the scenario users care about most. In the past years, several challenging GUI grounding benchmarks [ 14 , 15 , 16 , 17 ] have emerged. However, the challenges these benchmarks emphasize do not align with those CUAs face in real-world settings. Existing benchmarks are often difficult because they involve rare high-resolution interfaces or require substantial software-specific knowledge [ 15 , 16 ] ; yet their tasks are typically limited to single-click actions, and their targets are primarily GUI widgets. In contrast, our empirical observations show that CUAs frequently need to operate on objects such as tables, documents, charts, and images, often through more complex interactions including dragging and drawing [ 16 ] . This mismatch has, in turn, influenced the direction of model development [ 10 , 11 , 12 , 18 , 19 , 20 , 21 , 22 , 23 ] : as shown in the Figure 2 , the failure rate for complex interactions is far higher than that for simple clicking. We therefore identify two major bottlenecks in the current development of GUI-based CUAs: the lack of benchmarks for evaluating complex operations and the lack of large-scale datasets for such interactions. To address these issues, we first manually construct CUActSpot , a benchmark that covers a broad set of mouse-based actions that are common in computer-use workflows. It spans five modalities: GUI, Text, Table, Canvas, and Natural Image, and includes not only clicking, but also dragging and drawing actions, such as tracing object boundaries in Photoshop for image cutout. We find that performance on CUActSpot differs substantially from conventional GUI grounding benchmarks [ 14 , 15 , 16 , 24 ] , while showing closer agreement with end-to-end agentic results such as OSWorld [ 17 ] . This suggests CUActSpot may better reflect real-world computer-use scenarios. We further propose a data synthesis pipeline that obtains screenshots and coordinate-related metadata through code-based rendering, and we find that advanced GPT models can be leveraged to synthesize data for complex operations. Using this approach, we generate 50M samples that can support model pre-training or mid-training. We conduct ablation studies and empirical analyses over different data compositions and derive several insights. For example, we observe that, compared with simply scaling the amount of training data within a single modality, increasing data diversity substantially improves the model’s general interactive capability, a phenomenon we term variety scaling . Finally, our trained and open-sourced Phi-Ground-Any-4B achieves state-of-the-art performance among grounding models below 32B parameters. We hope that the benchmark, model, data, and insights presented in this paper will be valuable to the community and the broader industry. 2 Related Works Computer Use Agents Computer-use agents (CUAs) perceive screens and perform actions (e.g., clicks and keystrokes) to complete tasks autonomously. CUA development follows two paradigms. Modular CUAs pair a frontier VLM as a planner with a dedicated grounding model for precise low-level actions (e.g., UGround [ 10 ] , SeeClick [ 14 ] , OS-Atlas [ 11 ] ), though the natural-language interface between them can lose spatial and contextual information. End-to-end CUAs unify perception, reasoning, and action grounding within a single model, enabling joint optimization at the cost of massive training data. Commercial products such as Claude Computer Use [ 1 ] and OpenAI CUA [ 2 ] have brought this paradigm to end users, while open-source models including UI-TARS [ 13 ] , OpenCUA [ 25 ] MAI-UI [ 22 ] and EvoCUA [ 26 ] have rapidly approached comparable performance. However, a substantial gap between CUAs and human performance persists in complex scenarios such as document editing or multi-application coordination [ 17 ] . A key contributor is action grounding. GUI Action Grounding. GUI action grounding refers to localizing a target position on screen given a natural-language instruction, serving