UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu · 2025
Read UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis by Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11257.txt.
Introduction
Xinyi Liu and co-authors build a large synthetic dataset to teach AI agents where to click in app interfaces, in a 2025 paper. Vision-based agents must map a spoken instruction to the right on-screen element. Training data for that skill are scarce and costly to label by hand. Skewed element sizes and types make learning harder still.
Their pipeline, UI-E2I-Synth, generates instruction data at scale. It varies complexity on purpose, from plain labels to implied requests. Element size ratios and type balance are handled by design. Agents trained with such data should ground instructions more reliably across apps. Better grounding is a base need for software that operates computers on a user's behalf.
Catalog record
| CREATOR | Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11257
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu Abstract Recent advancements in Large Vision-Language Models are accelerating the development of Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities to enhance productivity on digital devices. Compared to approaches predicated on GUI metadata, which are platform-dependent and vulnerable to implementation variations, vision-based approaches offer broader applicability. In this vision-based paradigm, the GUI instruction grounding, which maps user instruction to the location of corresponding element on the given screenshot, remains a critical challenge, particularly due to limited public training dataset and resource-intensive manual instruction data annotation. In this paper, we delve into unexplored challenges in this task including element-to-screen ratio, unbalanced element type, and implicit instruction. To address these challenges, we introduce a large-scale data synthesis pipeline UI-E2I-Synth for generating varying complex instruction datasets using GPT-4o instead of human annotators. Furthermore, we propose a new GUI instruction grounding benchmark UI-I2E-Bench , which is designed to address the limitations of existing benchmarks by incorporating diverse annotation aspects. Our model, trained on the synthesized data, achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline. The proposed benchmark, accompanied by extensive analyses, provides practical insights for future research in GUI grounding. We will release corresponding artifacts here . † † † Equal contribution. ∗ Work done during the internship at Microsoft Research Asia. Figure 1: GUI grounding requires to localize elements in screenshots based on user instructions, with implicit instructions posing greater reasoning challenges than explicit ones. Figure 2: Left: Existing benchmark’s element-to-screen ratio significantly deviates from typical real-world desktop displays like 1080p and 1440p, on landscape samples. Right: The comparison shows Text and Icon dominates existing benchmark, leaving the rest of element types omitted. . 1 Introduction Graphical User Interface (GUI) agents are designed to understand and automate human instruction in the digital devices, promising a future where humans are liberated from repetitive tasks, thereby enhancing operational efficiency and productivity. The rise of Large Language Models (LLMs) Achiam et al. (2023) made it feasible to generally understand human instructions and structured text data. Hence, early efforts Gur et al. (2023) ; Zhou et al. (2023) ; Shaw et al. (2023) ; Deng et al. (2024) are made to operate GUI by understanding the text-rich metadata behind GUI, e.g. , HTML. However, some research Fu et al. (2024) ; Zheng et al. (2024) ; WebAIM (2024) has shown that GUI metadata is vulnerable and highly dependent on application developers for specific platforms. To overcome these limitations, more recent research Zhang et al. (2023b) ; Cheng et al. (2024a) ; Gou et al. (2024) ; Wu et al. (2024) ; Lin et al. () has concentrated on training Vision-Language Models (VLMs) to mimic human behavior in GUI operations, manipulating GUI elements from pixels based on user instructions. This approach makes GUI grounding capability as the core, which maps the user instruction into the corresponding element by outputting specific coordinates. Despite these works show satisfying progress on existing benchmark Cheng et al. (2024b) . Our investigation shows that there still is big gap between the existing benchmark and complicated and various GUI grounding environments. In this work, we thoroughly study the whole progress to build an element grounding dataset from screenshot collection to final instruction generation, revealing three main aspects that have not been fully explored previously: (a) Element-to-screen ratio : GUI instruction grounding necessitates higher resolution and smaller elements compared to natural scene object grounding. While previous studies Cheng et al. (2024b) ; Gou et al. (2024) have discussed the screenshot resolution in training dataset, we argue that the key factor is the element-to-screen ratio, i.e. , the size of an element relative to the screenshot. This ratio is affected by both resolution and UI zoom level. As shown in the left of Figure 2 , the landscape samples in existing benchmark has a larger element-to-screen ratio than typical real-world desktop displays like 1080p and 1440p, potentially leading to an overestimation of model performance. (b) Unbalanced element type : Different GUI element types exhibit diverse appearances and interaction designs, also with varying frequencies of occurrence. For instance, text buttons represent the most prevalent category, whereas checkboxes are comparatively less common. Unlike text buttons, whose functionality is often explicitly conveyed by their textual labels, checkboxes typically rely on surrounding elements to define their purpose. This distinction has not been adequately addressed in previous research. (c) Implicit instruction : In the context of GUI instruction grounding, users could convey their instructions based on their own intuitive understanding of element functionality or position, causing lack of direct correspondence with visible text on the screenshot, which we refereed as “implicit instruction”. Figure 2 showcases the complexity of implicit instruction. The implicit instruction challenges the understanding and reasoning capability of VLMs. To address the aforementioned challenges, we introduce a large-scale instruction grounding data synthesis pipeline, termed E lement-to- I nstruction Synth esis, abbreviated as UI-E2I-Synth . This pipeline utilizes GPT-4o instead of human annotators to synthesize realistic grounding user instructions of varying complexity, which differs from the element referring expressions that previous works curated. By the principle of divide and conquer, UI-E2I-Synth decomposes grounding instruction synthesis into three subtasks and execute them step by step. Initially, we collect screenshot-metadata pairs from various sources at different resolutions, then using a heuristic GUI metadata parser to extract reliable element attributes. Next, With these high-quality attributes to alleviate hallucinations, both explicit and implicit referring expressions are prompted to generate from GPT-4o. In the final step, GPT-4o is employed again to simulate user behavior by generating specific action parameters. These action parameters are then combined with the REs to synthesize final realistic user instructions. To more comprehensively evaluate model performance on GUI instruction grounding, we introduce a new benchmark UI-I2E-Bench that includes detailed annotations curated through a combination of our synthesis pipeline and human annotators. These annotations encompass various dimensions such as element type, element-to-screen ratio, and the level of instruction implicitness. Through rigorous evaluation on our proposed benchmark, we demonstrate that existing VLMs are still far from being satisfactory. To address the gap, we apply the proposed UI-E2I-Synth to collect a synthesized GUI grounding dataset comprising 9.9M instructions. Then following OS-Atlas Wu et al. (2024) we fine-tune two different pre-trained VLMs on the curated dataset and evaluate them. The experimental results demonstrate that our models achieve advanced performance on both the existing benchmarks and our proposed one, with substantially less instruction data. Notably, our models show outstanding capabilities in comprehending implicit instructions and handling long-tailed element types. These findings validate the efficacy of our data synthesis framework for GUI grounding, while our comprehensive benchmark analysis provides practical insights for future research in this domain. 2 Related work Autonomous GUI Agent and Vison-Language Models. With the development of LLMs, researchers have built autonomous agents to perform complex reasoning tasks Sumers et al. () ; Yang et al. (2024) ; Xi et al. (2025) . Early works Gur et al. (2023) ; Zhou et al. (2023) ; Shaw et al. (2023) ; Deng et al. (2024) have initiated the development of GUI agent frameworks for webpages and mobile platforms using