MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
SeokJoo Kwak, Jihoon Kim, Boyoun Kim, Jung Jae Yoon, Wooseok Jang, Jeonghoon Hong, Jaeho Yang, Yeong-Dae Kwon · 2025
Read MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements by SeokJoo Kwak, Jihoon Kim, Boyoun Kim, Jung Jae Yoon, Wooseok Jang, Jeonghoon Hong, Jaeho Yang, Yeong-Dae Kwon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13087.txt.
Introduction
SeokJoo Kwak and seven co-authors improve how AI agents find things on a screen. Grounding, turning an instruction into exact screen coordinates, is essential for software agents and accessibility tools. Single models handling the whole screen at once lose accuracy in clutter and ambiguity. MEGA-GUI splits the job: specialized vision-language agents first pick a rough region of interest, then ground the element inside it. A bidirectional zoom fights the dilution that comes from shrinking a big screen, and a rewriting agent clarifies vague instructions before the search begins.
Different vision-language models proved strong at different visual scales, which the pipeline exploits. On ScreenSpot-Pro, a dense professional-screen benchmark, the system reached about seventy-three percent accuracy, and on the semantically harder OSWorld-G it reached about sixty-nine percent, both above earlier results. The authors release code and a grounding benchmark toolkit.
Catalog record
| CREATOR | SeokJoo Kwak, Jihoon Kim, Boyoun Kim, Jung Jae Yoon, Wooseok Jang, Jeonghoon Hong, Jaeho Yang, Yeong-Dae Kwon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements SeokJoo Kwak, Jihoon Kim, Boyoun Kim, Jung Jae Yoon, Wooseok Jang, Jeonghoon Hong, Jaeho Yang, Yeong-Dae Kwon Abstract Graphical User Interface (GUI) grounding—the task of mapping natural language instructions to screen coordinates—is critical for autonomous agents and accessibility technologies. Prior methods, often built on monolithic models or one-shot, unidirectional pipelines, lack the modularity and error recovery needed to handle visual clutter and ambiguous instructions. We introduce MEGA-GUI , a multi-stage framework that decomposes GUI grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrating specialized agents based on diverse Vision-Language Models (VLMs). Central to this framework is a bidirectional ROI zoom algorithm that alleviates spatial dilution, complemented by a context-aware rewriting agent that reduces semantic ambiguity. Through analysis, we show that VLMs possess complementary strengths and weaknesses depending on the input’s visual scale and that leveraging these differences with a modular design substantially outperforms any monolithic approach. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI achieves 73.18% accuracy; on the semantically complex OSWorld-G benchmark, it scores 68.63%, significantly surpassing prior results. To support reproducibility and future research, we release all code and the Grounding Benchmark Toolkit (GBT) at: https://github.com/samsungsds-research-papers/mega-gui . 1 INTRODUCTION Recent breakthroughs in multimodal AI, exemplified by agents leveraging OpenAI’s Computer Use API with GPT-4o ( 12 ) and Anthropic’s screen-aware Claude models ( 1 ) , have opened a new frontier in autonomous computer control. These systems promise to revolutionize domains ranging from enterprise software automation to personal assistive technologies. However, their practical effectiveness remains fundamentally limited by a persistent and critical challenge: GUI grounding—the task of accurately mapping natural language commands to specific screen coordinates. Despite significant advances, current agents still exhibit high error rates on complex grounding benchmarks, a critical limitation documented across numerous studies ( 20 ; 8 ; 21 ) rendering them unreliable for most real-world scenarios. This difficulty stems from a dual challenge. First, the proliferation of high-resolution displays leads to extreme spatial dilution, creating a “needle-in-a-haystack” problem where target widgets may occupy less than 0.1% of the total screen area ( 20 ) . This saturates a model’s perceptual capacity with distractors, making accurate localization exceedingly difficult. Second, user instructions are often semantically ambiguous. Commands like “click the small wrench icon” or “close the dialog” require contextual reasoning that goes beyond simple visual pattern matching ( 10 ) . Failures to address these intertwined issues in cluttered, real-world environments underscore the urgent need for grounding methodologies that are both resilient and precise. To address this, the dominant approach has been to train large, monolithic Vision-Language Models (VLMs) on massive, annotated GUI datasets ( 24 ; 2 ) . While these models have pushed the field forward, they treat grounding as a single, end-to-end task, which can limit their adaptability. This limitation mirrors a broader paradigm shift within the AI community; the research focus is evolving from enhancing the performance of monolithic models on single-query tasks towards designing and orchestrating multi-agent systems to solve complex, multi-step problems ( 18 ) . This evolution presents a clear opportunity to reframe GUI grounding—not as a monolithic prediction problem, but as a structured control task. Building on this, a second strategy explores agentic, inference-time refinement pipelines that iteratively narrow the region of interest ( 11 ; 8 ) . While this marks progress toward agentic control, these methods remain fundamentally fragile: their unidirectional refinement paths lack mechanisms for backtracking or context recovery, so early errors cannot be corrected. In addition, such frameworks are often assembled in an ad-hoc fashion, hard-wiring a single VLM into a fixed pipeline and overlooking a crucial insight: the sub-tasks of grounding—broad contextual search and fine-grained element identification—differ substantially in nature and place very different demands on the system. In this paper, we introduce MEGA-GUI , a modular framework that reformulates GUI element grounding as a structured, multi-stage control problem. By orchestrating powerful VLMs as specialized “tools” within a stateful graph, MEGA-GUI achieves new levels of robustness and precision. Our work contributes a novel methodology specifically tailored to the GUI grounding task, moving beyond ad-hoc system integration toward principled design. Its primary contributions are threefold: 1. A Principled, Decoupled Framework: We propose a novel architecture that separates coarse spatial localization from fine-grained semantic grounding. Its centerpiece is a robust bidirectional adaptive zoom algorithm, a recoverable search mechanism that uses geometric feedback to proactively refine the agent’s visual focus and recover from initial prediction errors. 2. Empirical Validation of a “No Free Lunch” Principle: MEGA-GUI establishes new state-of-the-art performance on the ScreenSpot-Pro and OSWorld-G benchmarks. We provide the first empirical validation that a “No Free Lunch” principle applies to VLM specialization, grounded in a systematic study of model performance across a spectrum of visual scales (window sizes). Our analysis reveals that different models possess distinct, scale-dependent strengths for coarse visual search versus precise, fine-grained identification, providing a principled, data-driven basis for our modular agent design. 3. An Open-Source Toolkit for Principled Design: We release our Grounding Benchmark Toolkit (GBT) , a comprehensive suite for the systematic evaluation of GUI grounding components. By enabling a data-driven methodology for agent design, the GBT fosters more reproducible, transparent, and rigorous research, aligning with the community’s growing emphasis on responsible AI development. 2 RELATED WORK Early GUI automation relied on structured metadata like accessibility trees ( 9 ; 16 ; 5 ) , but the prevalence of custom-rendered widgets in modern applications has necessitated a shift toward vision-based agents that operate directly on raw pixels. These vision-centric agents have evolved along several primary architectural paradigms, each making significant contributions to the field. General Purpose Vision-Language Models. Pre-trained VLMs such as CUA ( 13 ) , Gemini ( 17 ) , and Qwen ( 15 ) represent a foundational pillar in this area. Trained on vast datasets of image-text pairs ( 4 ) , these models possess remarkable general visual understanding, enabling them to interpret a wide array of GUI elements. Their versatility provides a powerful starting point for numerous applications, including interface navigation and user intent understanding, establishing a robust baseline upon which more specialized systems can be built. Specialized Models for GUI Grounding. Building upon this foundation, a key research direction involves creating specialized agents through fine-tuning, with a clear evolution in both the data and the methods used for training. One line of work focuses on the data source: models like SeeClick ( 3 ) and UGround ( 6 ) leverage scalable but sometimes noisy web data , while UI-TARS ( 14 ) uses high-precision manual annotations, and JEDI ( 21 ) pioneers large-scale data synthesis to combine the benefits of both. A parallel evolution is occurring in training methodology. While most early models relied on Supervised Fine-Tuning (SFT), recent models like GTA1 ( 23 ) have introduced reinforcement learning (RL) to the grounding task. This approach moves beyond predicting a single center point by rewarding any correct click within a target element, better aligning the training objective with the real-world success condition. Integrated Refinement Pipelines. A third paradigm focuses on enhancing the zero-shot performance of VLMs through innovative inference-time strategies. These methods have pioneered the use of iterative refinement of a Region of Interest (ROI)