Grounding Computer Use Agents on Human Demonstrations
Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar · 2025
Read Grounding Computer Use Agents on Human Demonstrations by Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07332.txt.
Introduction
Aarash Feizi and sixteen co-authors build the missing training data for AI agents that operate desktop computers. Such agents must ground language instructions in the correct on-screen elements, but while web and mobile grounding data sets abound, desktop resources are scarce. GroundCUA is a large-scale dataset built from expert human demonstrations across eighty-seven applications in twelve categories, with fifty-six thousand screenshots and over three and a half million human-verified annotations.
From the demonstrations the team generates diverse instructions reflecting real tasks, then trains the GroundNext model family on them. At three- and seven-billion-parameter scales, GroundNext set state-of-the-art results on five benchmarks while using less than a tenth of the training data of prior work. Reinforcement learning post-training helped further, and in an agentic OSWorld evaluation the models matched or beat systems trained with far more data.
Catalog record
| CREATOR | Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Grounding Computer Use Agents on Human Demonstrations Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar Abstract Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA , a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56 3.56 M human-verified annotations. From these demonstrations, we generate diverse instructions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GroundCUA , we develop the GroundNext family of models that map instructions to their target UI elements. At both 3B and 7B scales, GroundNext achieves state-of-the-art results across five benchmarks using supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GroundNext attains comparable or superior results to models trained with substantially more data,. These results demonstrate the critical role of high-quality, expert-driven datasets in advancing general-purpose computer-use agents. † † footnotetext: * Equal contribution. † Equal supervision. Project Page: https://groundcua.github.io 1 Introduction The vision of computer-use agents (CUA) that operate software on behalf of users has gained significant momentum with recent progress in multimodal large language model–based agents ( OpenAI 2025 ; Anthropic 2024a ; Qin et al. 2025 ; Wang et al. 2025a ) . These agents promise to automate routine work and make complex digital tools more accessible. For such agents to succeed, they must first plan the next step in a task, then ground the plan to the exact on-screen element to click, type, or drag. Accurate grounding is critical: without correctly identifying the right button or menu item, even a flawless plan cannot be executed. In FreeCAD, for instance, when asked to “open the color picker” (Figure 1 ), the agent must distinguish a small palette icon from look-alike tools, one of which it must precisely click. When grounding fails, the plan quickly veers off course, minor errors compound, and tasks ultimately fail ( Nayak et al. 2025 ) . Moreover, grounding in desktop applications is challenging due to their complexity and diversity. These applications often feature high-resolution displays with dense layouts and visually similar elements, making precise localization difficult. Additionally, desktop applications can contain user-specific artifacts (e.g., documents or spreadsheets) that may not have been seen during training, adding variability and unseen contexts. Finally, collecting automated datasets for desktop environments with strong coverage is also challenging, as highlighted by recent datasets ( Gou et al. 2024b ; Wu et al. 2024 ; Xie et al. 2025 ) . Figure 1: Overview of the GroundCUA dataset and GroundNext models. Human demonstrations of computer-use tasks are recorded as screenshots (example from FreeCAD) with UI metadata, which are processed into high-quality natural language instruction tasks for UI grounding. GroundNext is trained in two stages: SFT (700K samples) followed by RL (10K samples), achieving state-of-the-art grounding performance with efficient training. To this end, we introduce GroundCUA , a large-scale, human-annotated dataset for desktop grounding. The dataset spans 87 applications across 12 categories, with 56 K screenshots and 3.56 M+ element annotations. These annotations are collected from task demonstrations by trained annotators, ensuring high-quality and densely labeled data that provides rich context for effective model training. It also reflects the pixel diversity of desktops, with resolutions ranging from 500K to 7M pixels and a substantial proportion of very small bounding boxes, highlighting the fine-grained challenges agents must overcome. Furthermore, GroundCUA includes fine-grained category information (menus, buttons, etc.) for 50% of the UI elements and includes multiple variants of related applications (e.g., LibreOffice and OnlyOffice), directly addressing the difficulty of similar yet distinct applications and enabling agents to learn robust, application-specific grounding strategies. Key highlights of GroundCUA compared to other datasets are: Scale: 56K annotated screenshots and 3.56 million elements; Resolution, Element Size, and Density: High-resolution images with maximum annotation density, covering almost every visible element, including small elements like icons and controls; Expert Quality: Human-verified annotations for high accuracy; Application Diversity: 87 desktop applications for broad real-world coverage. Using this dataset, we construct a 700K image-instruction pair instruction-tuning set that mimics real-world semantic interactions. We introduce the GroundNext series of vision-language models, designed for precise grounding across desktop applications. The series includes models at 3B and 7B scales, offering a balance between efficiency and accuracy. Each model is trained in two stages: first, supervised fine-tuning (SFT) on 700K curated datapoints from GroundCUA , and second, reinforcement learning (RL) to further refine performance. This approach enables GroundNext to achieve state-of-the-art results on key desktop benchmarks, including ScreenSpotPro ( Li et al. 2025 ) , OSWorld-G ( Xie et al. 2025 ) , and UI-Vision ( Nayak et al. 2025 ) . Despite using significantly fewer SFT datapoints than state-of-the-art models like JEDI (which are trained on 9M datapoints), GroundNext outperforms existing models, demonstrating its efficiency in training and proving that high-quality, well-curated data can outperform larger, less precise datasets. In the RL stage, GroundNext further refines its grounding accuracy, achieving significant improvements without relying on complex reward strategies , unlike many RL-tuned models, which typically incorporate specialized reward functions and additional objectives. This shows the effectiveness of combining supervised fine-tuning (SFT) with high-quality data. This efficiency is further highlighted in agentic, multi-step tasks; evaluated on the OSWorld-Verified benchmark, GroundNext -3B not only significantly outperforms its 3B peers but also surpasses many larger models, including OpenCUA-72B and proprietary APIs. Notably, our 3B model achieves performance comparable to the much larger JEDI-7B, demonstrating significant practical utility for real-world, resource-constrained systems. Additionally, GroundNext excels in cross-platform generalization , delivering strong performance across desktop, mobile, and web environments; even though we only train on desktop dataset. Evaluated on benchmarks like MMBench-GUI (L2) and ScreenSpot-v2, in addition to desktop-specific tasks, GroundNext showcases its ability to generalize across a wide range of user interfaces and platforms. In summary, our contributions are as follows: • We introduce GroundCUA , a large-scale, human-annotated desktop grounding dataset with over 3.56 million annotations across 56 K screenshots from 87 applications in 12 categories, providing dense, high-resolution, and fine-grained supervision for robust computer-use agents. • We present the GroundNext series, vision-language models at 3B and 7B scales, trained on GroundCUA with SFT and RL, achieving state-of-the-art performance across desktop benchmarks and multi-step agentic tasks, with significantly fewer datapoints than prior models. • We provide a comprehensive analysis of SFT and RL roles, evaluate our dataset’s cross-domain impact and generalization beyond desktop, and study the benefits of open-source software for grounding performance. We release both GroundCUA and the GroundNext models to support open research. 2 Related Work Computer-Use Agents. Recent advancements in computer-use agents have focused on enhancing their ability to understand and interact with user interfaces, ranging from simple commands to complex, multi-step