ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, Kang Min Yoo, KAIST, NAVER Cloud {hs.lee, jinwoos}@kaist.ac.kr , [email protected] · 2025
Read ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search by Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, Kang Min Yoo, KAIST, NAVER Cloud {hs.lee, jinwoos}@kaist.ac.kr , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15259.txt.
Introduction
Lee and nine co-authors present a study from twenty twenty-five on teaching agents to find screen elements with far less training data. Agents that use computers must locate buttons and fields exactly. Prior work leans on huge web datasets to raise that accuracy. Their framework learns through self-made reasoning steps, tuned by online reinforcement learning. A spatial critic checks guesses against flipped or shifted views of the screen. At test time, a spatial search plus vote step boosts results further.
The method tops grounding benchmarks while using a sliver of rival training data. One comparison uses only a tiny fraction of the best open baseline's samples. The authors stress data thrift over data scale. Their frame suits web agents where labeled data are costly.
Catalog record
| CREATOR | Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, Kang Min Yoo, KAIST, NAVER Cloud {hs.lee, jinwoos}@kaist.ac.kr , [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, Kang Min Yoo, KAIST, NAVER Cloud {hs.lee, jinwoos}@kaist.ac.kr , [email protected] Abstract Recent advances in Multimodal Large Language Models (MLLMs) have enabled autonomous agents to interact with computers via Graphical User Interfaces (GUIs), where accurately localizing the coordinates of interface elements (e.g., buttons) is often required for fine-grained actions. However, this remains significantly challenging, leading prior works to rely on large-scale web datasets to improve the grounding accuracy. In this work, we propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective framework for web grounding that enables MLLMs to learn data efficiently through self-generated reasoning and spatial-aware criticism. More specifically, ReGUIDE learns to (i) self-generate a language reasoning process for the localization via online reinforcement learning, and (ii) criticize the prediction using spatial priors that enforce equivariance under input transformations. At inference time, ReGUIDE further boosts performance through a test-time scaling strategy, which combines spatial search with coordinate aggregation. Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks, outperforming baselines with substantially fewer training data points (that is, only 0.2% samples compared to the best open-source baselines). 1 Introduction Graphical User Interface (GUI) agents—i.e., Multimodal Large Language Model (MLLM) agents that interpret visual screen contents and generate web actions in natural language to navigate the web environment—have shown promising capabilities in web navigation tasks as MLLMs continue to improve in general decision-making ability [ 26 , 3 , 28 , 41 ] . Despite the recent significant efforts, however, a substantial gap remains between the performance of these agents and that of proficient human users [ 6 ] , particularly in complex or long-horizon tasks [ 39 ] . This gap arises mainly from two challenges: limited visual understanding of web pages [ 33 ] and insufficient web domain-specific decision-making ability [ 11 ] . To address this, prior work typically decomposes the problem into two parts: (i) building an MLLM that can interpret the visual input [ 41 ] and (ii) employing a language model that performs decision-making and planning based on the MLLM’s interpretation [ 28 , 14 ] , where the major bottleneck lies in the visual understanding component [ 11 ] . Building an MLLM that can predict the exact pixel coordinates of target region on the screen (e.g., buttons) has shown great promise in providing the visual understanding required by another LLM for effective decision-making [ 11 ] . However, grounding remains challenging as it requires fine-grained skills such as reasoning and the manifestation of spatial understanding through token-based predictions [ 24 , 40 ] . Recent approaches have emphasized the importance of large-scale, high-quality image–instruction pairs for training MLLMs through supervised fine-tuning (SFT) [ 22 , 11 ] . Gou et al. [11] proposes synthesizing diverse referring expressions that can express the same object in several views. However, such SFT models depend on costly data curation to perform well, which poses a major scalability challenge [ 17 ] . To this end, we focus on extracting rich information from web image data to achieve data-efficient grounding by learning to explain the coordinate prediction process in natural language and by leveraging spatial priors to maximize the utility of visual input. Contribution. We propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective GUI coordinate grounding method for web agents. Specifically, ReGUIDE is composed of a two-stage training: (i) self-generation of image explanation through its own reasoning, and (ii) criticism of the localization prediction via spatial priors. First, ReGUIDE learns to generate the language description of the given web image that can guide itself to correctly predict the coordinate, where this prediction accuracy is used as the reward for online RL. Then, by leveraging a spatial prior—i.e., the fact that augmentations such as cropping lead to equivariant changes in the target coordinates—the model criticizes its predictions by ensuring consistent outputs under the same language description across augmented image–coordinate pairs. Furthermore, we introduce an inference-time scaling strategy for ReGUIDE, which integrates spatial search with an LLM test-time scaling scheme. Specifically, the model generates multiple localization predictions and crops the region where the target is likely to exist based on the predictions. In this region, we generate multiple coordinate candidates, then aggregate the candidates into a single coordinate via statistical voting strategy (i.e., Kernel Density Estimation [ 30 ] (KDE)-based voting). We demonstrate the effectiveness of ReGUIDE through evaluations on multiple web-grounding datasets and agent-setting benchmarks. Notably, ReGUIDE enhances web coordinate grounding performance beyond prior methods, achieving the state-of-the-art performance in web grounding. For instance, ReGUIDE improves the grounding accuracy of Qwen-2.5-VL-3B from 55.5% to 87.3% on Screenspot [ 7 ] and from 23.9% to 44.3% on Screenspot-Pro [ 20 ] . Furthermore, our experimental results show that ReGUIDE indeed guides the GUI agent to improve the overall decision-making ability, showing a significant performance improvement in agentic tasks. 2 Related Work Graphic User Interface (GUI) grounding. Recent advances in pixel-level GUI grounding have demonstrated mapping natural language instructions to screen coordinates without relying on HTML or DOM structures [ 31 ] . Several prior works [ 11 , 37 , 35 , 8 , 22 ] train on over a million synthesized screenshots and achieve superior performance. Gou et al. [11] further shows that synthesizing and augmenting text instructions for each image element can lead to further improvements. However, current approaches depend on massive annotated datasets, incurring substantial computation and labeling costs, and neither leverages the reasoning capabilities of large language models to improve localization under data-scarce or out-of-distribution conditions. Reinforcement learning in MLLM. Reinforcement learning has emerged as a powerful mechanism to fine-tune multimodal large language models via self-improvement and feedback. Self-Critical Sequence Training [ 29 ] pioneered policy-gradient optimization for image captioning. Building on this paradigm, RLHF-V [ 38 ] formulates multimodal RLHF under constrained optimization to jointly maximize helpfulness and minimize unsafe outputs. More recently, Vision-R1 [ 17 ] generates synthetic chain-of-thought trajectories and applies iterative policy optimization to boost multimodal math reasoning. While UI-R1 [ 23 ] and GUI-R1 [ 36 ] are concurrent works that solves the GUI grounding problem with reinforcement learning, ReGUIDE further explores the grounding by utilizing the spatial prior and test-time spatial search. As a result, ReGUIDE significantly outperforms previous models in grounding evaluations as shown in Appendix. Test-time scaling. Recent works explored that scaling test-time computation, such as best-of-N sampling, can be even better than scaling train-time computation for performance [ 32 ] . Specifically, test-time scaling strategies improve LLM performance by generating numerous candidate outputs and selecting the best [ 32 , 19 , 15 ] . To enhance decision-making, external verifiers are often employed to evaluate and refine these outputs [ 15 ] . In the localization task, recent work framed the localization as a search problem and suggested a test-time scalable strategy [ 34 ] . (a) Learning to explain web images via reasoning (b) Learning to predict consistent coordinates Figure 1 : Overview of training ReGUIDE. Left : The model rolls out multiple (reasoning,