GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu · 2026
Read GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents by Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09770.txt.
Introduction
Chen Chen and six co-authors give interface agents eyes that choose where to look. Their 2026 arXiv paper presents GUI-Eyes, a reinforcement learning framework for active visual perception in graphical interfaces. Vision-language models have advanced GUI automation, but most still take one static screenshot.
A passive agent cannot zoom, re-check, or look at the right moment. GUI-Eyes learns when, whether, and how to observe the screen. Active looking should ground clicks and typing in what is actually shown. That promises fewer blind errors in agents that book, buy, and fill forms.
Catalog record
| CREATOR | Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-09770
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu Abstract Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes the decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents. 1 Introduction The development of large language models (LLMs) ( 37 ; 10 ; 36 ; 49 ) and vision language models (VLMs) ( 41 ; 4 ; 1 ; 17 ) has introduced new opportunities and challenges in applying these models to GUI tasks. Existing GUI agents are mainly based on supervised fine-tuning (SFT) ( 47 ; 28 ; 14 ) , where large annotated datasets are used to teach model interface understanding and action planning. However, this approach suffers from several limitations: it requires expensive, labor-intensive data collection, and the resulting models often lack robustness when deployed in unfamiliar or out-of-domain environments ( 5 ; 27 ) . DeepSeek-R1 ( 12 ) demonstrates that reinforcement learning (RL) can enhance large models’ problem-solving ability without human-labeled data, by optimizing behavior through interaction and well-designed rewards. In GUI tasks, RL effectively reduces supervision demands while improving adaptability and generalization ( 26 ; 23 ; 25 ; 56 ; 9 ) , making it a promising alternative to SFT. Despite this potential, most existing methods still optimize only textual outputs ( 25 ; 26 ; 23 ; 51 ) , overlooking visual cues essential for GUI understanding. Real users rely on visual attention to locate elements and interpret layouts ( 54 ; 48 ; 14 ; 33 ) ; models limited to language reasoning struggle with ambiguous instructions and complex interfaces. Thus, GUI agents must integrate perception and decision-making—learning not only what to see but how often to observe—to support robust visual-grounded interactions. To address the limitations of supervised learning in terms of data annotation cost and generalization, we propose GUI-Eyes , a reinforcement learning framework centered on active perception. Unlike traditional GUI agents that rely on static, one-shot visual input, GUI-Eyes empowers the model to dynamically decide whether to invoke visual tools during reasoning, and to flexibly configure their parameters (e.g., crop region, zoom scale) for acquiring task-relevant observations step by step. By modeling visual perception as an optimizable policy, GUI-Eyes learns a perception–reasoning–perception loop that tightly coordinates visual observation with language-based decision-making. As shown in Figure 1 , GUI-Eyes-3B achieves superior performance on the ScreenSpot-Pro benchmark compared to existing models of similar or larger scales, validating the effectiveness of our tool-aware active perception framework. Figure 1: Performance Scaling of Multimodal UI Understanding Models on the ScreenSpot-Pro Benchmark. Our method achieves state-of-the-art performance. Our main contributions are summarized as follows: • We propose GUI-Eyes , a novel framework that integrates active perception into GUI agents. It enables the model to autonomously determine when and how to invoke visual tools during reasoning, achieving more precise and task-adaptive visual understanding. • To support the learning of effective active perception behaviors, we design a multi-factor reward function that provides structured supervision over format correctness, initial localization, and spatial coverage. This facilitates more stable and generalizable tool-use policy learning. • We conduct extensive experiments on the ScreenSpot-Pro benchmark. GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3,000 labeled samples , significantly outperforming both supervised and RL-based baselines, thereby demonstrating strong sample efficiency and robust generalization. 2 Related Work 2.1 GUI Agents With the growing capabilities of multimodal large language models (MLLMs) ( 1 ; 41 ; 4 ; 17 ) , GUI-based agents have become a central focus in human–computer interaction research ( 39 ; 38 ; 52 ; 35 ) . Existing work in this area can be broadly categorized into two main paradigms: structure-driven and vision-driven approaches. Structure-driven methods rely on structured representations such as HTML or DOM trees to parse and execute interface instructions ( 13 ; 18 ; 7 ; 55 ; 29 ) . These methods benefit from explicit symbolic semantics and direct access to internal interface states. Vision-driven methods , in contrast, operate directly on GUI screenshots, leveraging visual perception and language instructions for open-ended reasoning and grounding ( 53 ; 14 ; 50 ; 22 ; 45 ) . For instance, AppAgent ( 53 ) explores autonomous interactions in mobile applications, while CogAgent ( 14 ) uses high-resolution visual encoders for better UI understanding. OS-Atlas ( 47 ) proposes a unified action representation and pre-trains on 13M UI elements to enable cross-platform generalization. ScaleTrack ( 15 ) designs a backtracking-based task to enhance multi-step reasoning, training on 7.5M screenshots for joint grounding and planning. Despite strong empirical results, these methods mainly depend on supervised learning with static input–output pairs, limiting their ability to actively acquire perceptual information or adjust reasoning strategies during inference. Figure 2: Overview of the GUI-Eyes Framework. The top illustrates a rollout example with optional visual tool invocation, together with a tool-specific reward function that combines spatial proximity and region overlap relative to the ground-truth. The bottom depicts the progressive inference architecture and end-to-end training pipeline, where the two-stage decision process is guided by stage-specific prompts, and visual inputs are dynamically generated through previously applied visual tools. Figure 3: Inference Example of Tool-Augmented Reasoning with Cropping in a GUI Task. 2.2 Reinforcement Fine-Tuning Traditional supervised fine-tuning (SFT) for GUI agents requires extensive annotated data ( 47 ; 28 ; 15 ; 46 ) , making it costly and less generalizable. Recently, rule-based reinforcement learning (RL) has emerged as a more efficient alternative, especially in low-resource scenarios. DeepSeek-R1 ( 12 ) introduced this paradigm for large language models by employing predefined reward functions (e.g., symbolic correctness) to evaluate outputs without human feedback. This idea has been extended to GUI agents ( 25 ; 26 ; 56 ; 23 ; 9 ; 43 ; 11 ) , showing strong data efficiency. UI-R1 ( 25 ) first introduced rule-based RL for low-level GUI action prediction, achieving strong performance with only 130 mobile samples and substantially lowering data requirements. InfiGUI-R1 ( 23 ) proposed the Actor2Reasoner framework, encouraging agents to “think before acting and reflect after execution,” thereby improving their understanding and operation in complex UI layouts. Interestingly, GUI-G1 ( 56 )