Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems
Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, Yueting Zhuang · 2025
Read Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems by Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, Yueting Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06470.txt.
Introduction
Software that clicks buttons from written instructions must first find the right element on screen, a task called grounding. Fei Tang and co-authors note that current systems guess in one shot, without reasoning, and they stumble on nested, layered interfaces. Their framework, Focus, copies human thinking instead: a fast mode for easy cases and a slow, deliberate mode for hard ones, switching between them as complexity demands.
Focus works in stages, summarizing the interface, analyzing the relevant region closely, then predicting exact click coordinates. It reached top accuracy on two standard benchmarks while training on far less data than rivals and using a modest two-billion-parameter model. The gains were largest on the harder professional-interface benchmark. The authors argue dual-mode processing is the practical route to reliable interface agents.
Catalog record
| CREATOR | Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, Yueting Zhuang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, Yueting Zhuang Abstract Humans can flexibly switch between different modes of thinking based on task complexity: from rapid intuitive judgments to in-depth analytical understanding. However, current Graphical User Interface (GUI) grounding systems which locate interface elements based on natural language instructions rely solely on immediate prediction without reasoning, struggling to understand complex interface layouts with nested structures and hierarchical relationships, limiting their effectiveness on complex interfaces. Inspired by human dual-system cognition, we present Focus , a novel GUI grounding framework that combines fast prediction with systematic analysis. The framework dynamically switches between rapid and deliberate processing through an adaptive system switching based on task complexity, optimizing both efficiency and accuracy. Focus decomposes grounding into progressive stages: interface summarization, visual focused analysis, and precise coordinate prediction. This structured decomposition enables systematic understanding of both interface layouts and visual relationships. Extensive experiments show that Focus achieves state-of-the-art performance using only 300K of the training data with a 2B parameter model compared to existing approaches. Focus demonstrates superior performance particularly in complex GUI scenarios, achieving 77.4% average accuracy on ScreenSpot and 13.3% on the more challenging ScreenSpot-Pro. Our analysis reveals the effectiveness of this dual-system approach while demonstrating its potential for improving complex GUI interaction scenarios 1 1 1 https://github.com/sugarandgugu/Focus . 1 Introduction GUI interactions are fundamental to modern human-computer interaction, driving the development of GUI agents that can automate complex interface operations He et al. 2024 ; Hu et al. 2024 . A critical capability for these agents is GUI grounding: the ability to accurately locate and interpret interface elements based on natural language instructions. This foundational skill directly impacts an agent’s effectiveness in understanding interface semantics and executing operations across diverse scenarios Cheng et al. 2024 ; Yang et al. 2024 . Early GUI agents primarily relied on structured information such as XML or DOM trees for element localization Wang et al. 2024b ; Zhang et al. 2023 ; Li et al. 2024 . While this approach provided reliable structural information, it faced significant limitations in practical applications due to information redundancy and accessibility issues Gou et al. 2024 ; Cheng et al. 2024 . Recent advances in multimodal large language models (MLLMs) Bai et al. 2023 ; Lu et al. 2024 ; Ye et al. 2024 ; Chen et al. 2024b ; Liu et al. 2023 have enabled direct screenshot-based approaches for GUI grounding. For instance, SeeClick Cheng et al. 2024 leverages visual GUI pretraining to enhance element localization, while ShowUI Lin et al. 2024 incorporates advanced visual-language processing techniques. Figure 1: Comparison of GUI grounding approaches. (a) Fast grounding system ( e.g. , SeeClick Cheng et al. 2024 , ShowUI Lin et al. 2024 ) directly predicts target locations without explicit reasoning. (b) Our Focus framework introduces a dual-system approach combining fast grounding with deliberate analysis, dynamically switching between systems based on task complexity. However, current MLLM-based GUI grounding methods Cheng et al. 2024 ; Lin et al. 2024 ; Yang et al. 2024 predominantly follow a direct prediction paradigm, attempting to locate target elements immediately from screenshots without deeper analysis. This approach, while efficient, faces two critical limitations: (1) Limited Interface Understanding : These methods struggle with complex interfaces containing multiple windows, nested menus, and hierarchical structures, lacking systematic analysis of interface layouts and element relationships. (2) Insufficient Visual Analysis : While accurate element identification requires understanding multiple visual attributes ( e.g. , color, shape, position) and their contextual relationships, current approaches attempt to directly predict target locations like human’s fast system, without engaging the slow system’s deliberate analysis that is characteristic of human analytical thinking. The efficient grounding and deeper analysis reflect a fundamental characteristic of human cognition in task processing. Humans employ two distinct cognitive systems when interpreting visual interfaces: a rapid, intuitive system for simple tasks and a deliberate, analytical system for complex scenarios Kahneman 2011 ; Evans 2008 . Inspired by this, we propose Focus , a GUI grounding framework that combines fast grounding with deliberate analysis. As shown in Figure 1 , Focus preserves the efficiency of direct prediction while introducing a systematic analysis process that first comprehends the interface layout before localizing target elements. The framework dynamically switches between two system based on task complexity, enabling efficient handling of both simple and complex GUI interactions. To train Focus , we generate structured training data that decomposes GUI grounding into three progressive stages: (1) interface summarization to capture overall layout structure, (2) focused analysis of relevant interface regions and their visual characteristics, and (3) precise coordinate prediction. This structured decomposition allows Focus to build comprehensive understanding from global context to local details. Through training on this multi-stage reasoning process, Focus develops robust generalization capabilities across diverse GUI scenarios. Our main contributions are as follows: • We propose Focus , a novel framework combining fast grounding with deliberate analysis for robust GUI grounding, drawing inspiration from human cognitive dual-process theory. • We introduce a hierarchical training approach that decomposes GUI grounding into progressive stages, enabling systematic interface understanding and visual analysis. • We develop an adaptive switching mechanism between fast and deliberate processing based on task complexity, optimizing both efficiency and accuracy. • Extensive experiments demonstrate Focus achieves state-of-the-art performance, with 1.4% and 17.7% improvements on ScreenSpot and ScreenSpot-Pro benchmarks respectively. 2 Method Focus enhances GUI grounding through dynamically adapting its processing strategy between fast and slow systems based on task complexity. For simple tasks, it employs fast grounding system for efficient processing. When facing complex scenarios, it activates slow grounding system, which generates an a task-oriented interface summarization and conducts focused analysis guided by the task instruction. To train such an adaptive dual-system, as shown in Figure 2 , we propose a data synthesis pipeline that decomposes GUI interactions into three progressive stages, constructing a 300K-sample dataset that captures human-like cognitive processes. Our approach includes three main components: (1) Dual-System Data Synthesis ( section 2.1 ) that constructs training data by decomposing GUI grounding into progressive stages, mimicking human’s transition from rapid intuitive judgments to in-depth analytical thinking; (2) Dual-System FOCUS Training ( section 2.2 ) that develops specialized model capabilities for both fast grounding and systematic analytical processing; (3) Adaptive System Switching ( section 2.3 ) that dynamically transitions between fast and slow systems based on task complexity. We detail our framework in the following sections. 2.1 Dual-System Data Synthesis To effectively train our dual-system model, we propose a progressive data synthesis strategy that systematically constructs training examples with increasing complexity while naturally differentiating between fast and slow cognitive processes. Our approach not only synthesizes training data but also adaptively separates examples based on their inherent complexity levels: We begin with fast grounding data synthesis, employing ShowUI Lin et al. 2024 for direct coordinate prediction. Successfully predicted cases form our fast grounding data, capturing straightforward tasks that can be solved through immediate perception. When initial predictions fail, the second stage enhances the process by incorporating interface summarization before prediction. The successful cases in this stage contribute to our slow grounding data, representing