Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
Yiqin Wang, Haoji ZhangJingqi Tian, Yansong TangShenzhen International Graduate School, Tsinghua University {yq-wang23@mails.,tang.yansong@sz.}tsinghua.edu.cn · 2024
Read Ponder & Press: Advancing Visual GUI Agent towards General Computer Control by Yiqin Wang, Haoji ZhangJingqi Tian, Yansong TangShenzhen International Graduate School, Tsinghua University {yq-wang23@mails.,tang.yansong@sz.}tsinghua.edu.cn (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01268.txt.
Introduction
Software agents that click and type for us mostly read page code, not the screen. Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang build one in 2024 that works by sight alone, as folk do. (The catalog's author field is laced with school and mail text; the head byline is clean.) Their frame splits the job in two. A broad model reads the task and spells out the next step in words.
A second model, trained on screens, finds the exact spot to press. That finder beat past models by a wide mark on a known screen-spot test. Full runs on web, desk, and phone tasks also topped the field. The team say sight-only agents can span far more software than code-fed ones.
Catalog record
| CREATOR | Yiqin Wang, Haoji ZhangJingqi Tian, Yansong TangShenzhen International Graduate School, Tsinghua University {yq-wang23@mails.,tang.yansong@sz.}tsinghua.edu.cn |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control Yiqin Wang, Haoji ZhangJingqi Tian, Yansong TangShenzhen International Graduate School, Tsinghua University {yq-wang23@mails.,tang.yansong@sz.}tsinghua.edu.cn Abstract Most existing GUI agents typically depend on non-vision inputs like HTML source code or accessibility trees, limiting their flexibility across diverse software environments and platforms. Current multimodal large language models (MLLMs), which excel at using vision to ground real-world objects, offer a potential alternative. However, they often struggle with accurately localizing GUI elements—a critical requirement for effective GUI automation—due to the semantic gap between real-world objects and GUI elements. In this work, we introduce Ponder & Press, a divide-and-conquer framework for general computer control using only visual input. Our approach combines an general-purpose MLLM as an ‘interpreter’, responsible for translating high-level user instructions into detailed action descriptions, with a GUI-specific MLLM as a ‘locator’ that precisely locates GUI elements for action placement. By leveraging a purely visual input, our agent offers a versatile, human-like interaction paradigm applicable to a wide range of applications. Ponder & Press locator outperforms existing models by +22.5% on the ScreenSpot GUI grounding benchmark. Both offline and interactive agent benchmarks across various GUI environments—including web pages, desktop software, and mobile UIs—demonstrate that Ponder & Press framework achieves state-of-the-art performance, highlighting the potential of visual GUI agents. Refer to the project homepage here . $*$ $*$ footnotetext: Equal contribution. †Correspondence to Yansong Tang. 1 Introduction Researchers have long pursued the development of autonomous agents to assist humans in interacting with diverse GUI devices Shi et al. 2017 ; Yao et al. 2022 ; Li et al. 2020b . With recent advancements in Large Language Models (LLMs) Brown et al. 2020 ; Achiam et al. 2023 ; Touvron et al. 2023 ; Anthropic 2024 , agents for web browsing Gur et al. 2024 , office automation Wu et al. 2024 ; Tan et al. 2024 , and mobile apps Rawles et al. 2024b have been proposed to streamline user interactions and enhance productivity. Major technology companies have also contributed to this development by creating agents that facilitate user experiences, such as Apple Siri, Microsoft 365 Copilot, and Capcut smart video editor. Despite these advancements, existing GUI automation approaches face limitations in generalizability and adaptability across software environments. First, software-specific agents from tech companies often operate beneath the GUI layer, bypassing user-facing elements and thus sacrificing generalization by interacting directly with underlying code. Second, most GUI agents Shi et al. 2017 ; Humphreys et al. 2022 ; Gur et al. 2024 ; Yao et al. 2022 ; Li et al. 2020b ; Zhou et al. 2023 ; Deng et al. 2024 developed by the research community rely on supplementary information such as HTML, DOM, or accessibility trees, which makes them specific to certain platforms and software environments. Human interaction with GUIs, by contrast, relies exclusively on visual input and interaction through actions like mouse clicks, keyboard input, and screen taps. Therefore, a robust GUI agent designed for broad applicability shall ideally be able to operate using only visual input, similar to human perception, and output actions in a human-like manner. Developing vision-only general agents capable of human-like interactions with GUIs presents significant challenges. These challenges can be summarized as: • Task Decomposition: Interpreting and breaking down high-level task instructions into a series of executable actions within a software environment, ensuring that the GUI agent executes the correct action. • Precise GUI Localization: Accurately localizing GUI elements to facilitate the correct placement of actions, such as clicks or text inputs. Figure 1 : Different types of frameworks for vision-based GUI agents. Figure 2 : Ponder&Press improves vision-based GUI agent on a broad range of tasks. As shown in Figure 2 (a) and (b), previous efforts have sought to build end-to-end models that address both challenges simultaneously. High-level user instructions are directly mapped to action types, action values, and pixel coordinates in a single inference. However, this approach struggles due to the significant difference between the textual nature of actions and values, and the numerical nature of pixel coordinates. As shown in Figure 2 (a), general-purpose end-to-end multimodal models (MLLMs) Achiam et al. 2023 ; Anthropic 2024 ; Wang et al. 2024 often suffer from poor grounding performance. As shown in Figure 2 (b), GUI-specific models Cheng et al. 2024 , though specialized to GUI grounding, struggle with effectively decomposing complex user instructions. As a result, this types of models suffer from poor accuracy in predicting the action type (e.g. TYPE or CLICK) and action value (e.g. the typed content). The claims made above are proved in the experiment section. In this paper, we introduce a divide-and-conquer framework called Ponder & Press. It follows the design presented in Figure 2 (c), leveraging the user-instruction interpretation ability of general purpose MLLM, as well as the grounding ability of GUI-specific MLLM. As further shown in Figure 3 , the framework is composed of two distinct stages that dealt with the two challenges separately: (1) The ‘Ponder’ stage , involves an Instruction Interpreter that converts high-level user goals into executable steps. For instance, as shown in Figure 3 , when tasked with finding the stock price of ‘Netflix’ on Google Finance, the interpreter outputs: ‘To find the latest price of Netflix stock, I need to search for Netflix in the Google Finance platform. The search bar is visible at the top of the page, so I’ll use that to enter [Netflix]‘, along with a structured output ”Action: TYPE, Value: ‘Netflix’, Element Description: ‘Search bar with placeholder text [Search for stocks, ETFs & more]’”. This stage leverages the commonsense knowledge embedded in MLLMs to bridge the gap between high-level user instructions and textual, structured action descriptions. (2) The ‘Press’ stage , where we train a Visual Element Locator to map the ‘Element Description’ to pixel coordinates, requiring only a small labeled dataset while achieving state-of-the-art performance. Figure 3 : The framework of Ponder&Press agent. The framework consists of two core components: an Instruction Interpreter that translates high-level user instructions into actionable steps, and a Visual Element Locator that localizes GUI elements for interactions such as clicking or typing. Our method ensures that complex instructions can be decomposed and precisely executed within diverse GUIs. This modular design allows the agent to accurately understand user intent and execute precise actions Liu et al. 2025 , maintaining flexibility for general software control. Furthermore, by relying solely on visual inputs—without the need for HTML, accessibility trees, or other supplementary data—our purely visual GUI agent enhances generalizability across various platforms, avoiding the need for software-specific modifications. Our main contributions are as follows: • We propose Ponder & Press, a divide-and-conquer GUI agent framework that only relies on visual input to mimic human-like interaction with GUIs. It guarantees the generalizability across diverse environments. • We evaluate Ponder & Press locator on the GUI grounding benchmark ScreenSpot , outperforming previous state-of-the-art model by +22.5%. • We further conducted extensive evaluations of our framework on 4 widely used GUI agent benchmarks, demonstrating the effectiveness of our agent in offline, online, desktop, webpage, and mobile settings. 2 Related Work 2.1 Autonomous Agents for GUI Devices System-specific agents developed by technology companies are typically integrated beneath the GUI layer and