Visual Test-time Scaling for GUI Agent Grounding
Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research · 2025
Read Visual Test-time Scaling for GUI Agent Grounding by Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00684.txt.
Introduction
Tiange Luo and co-authors brought in RegionFocus in 2025, a method that helps software agents operate graphical links. An agent looking at a screenshot must pick the right button or field among hundreds of parts, and clutter makes that grounding step error-prone. Their idea borrows from how people lean in: spend extra effort at choice time by zooming into the region that matters.
RegionFocus dynamically enlarges related regions, cutting background clutter. It also keeps an image-as-map record that marks key landmarks at each step. That record leaves a transparent trail the agent can use to choose among likely actions. Even with a simple region picker. The method lifted scores sharply on two public benchmarks when layered onto two leading open vision-language agents, and set a new best grounding result on the harder professional-screen benchmark.
Catalog record
| CREATOR | Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Test-time Scaling for GUI Agent Grounding Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research Abstract We introduce RegionFocus, a visual test-time scaling approach for Vision Language Model Agents. Understanding webpages is challenging due to the visual complexity of GUI images and the large number of interface elements, making accurate action selection difficult. Our approach dynamically zooms in on relevant regions, reducing background clutter and improving grounding accuracy. To support this process, we propose an image-as-map mechanism that visualizes key landmarks at each step, providing a transparent action record and enables the agent to effectively choose among action candidates. Even with a simple region selection strategy, we observe significant performance gains of 28+% on Screenspot-pro and 24+% on WebVoyager benchmarks on top of two state-of-the-art open vision language model agents, UI-TARS and Qwen2.5-VL, highlighting the effectiveness of visual test-time scaling in interactive settings. We achieve a new state-of-the-art grounding performance of 61.6% on the ScreenSpot-Pro benchmark by applying RegionFocus to a Qwen2.5-VL-72B model. Our code will be released publicly at https://github.com/tiangeluo/RegionFocus . 1 Introduction Graphical user interface (GUI) agents have become increasingly pivotal in modern computing, powering applications ranging from automated web browsing to intuitive operating system navigation [ 1 , 21 ] . With the proliferation of large-scale vision-language models (VLMs), researchers have sought to harness both textual and visual information to build more capable interactive systems [ 2 ] . While many existing frameworks rely heavily on text-based reasoning [ 46 , 39 ] or simple visual grounding [ 22 , 11 ] , real-world GUIs often contain a substantial number of irrelevant elements—such as menu bars, ads, and extraneous buttons—that can overwhelm purely textual or naive visual approaches. This mismatch between text-heavy inference and the visual complexity of GUIs leads to frequent errors (e.g., clicking the wrong button or navigating to an unintended section). Since these tasks are typically high-level, such low-level mistakes accumulate and ultimately result in higher failure rates and poorer overall performance. Recent research on GUI agents typically falls into two main categories: those relying on textual cues for planning and reasoning, and those incorporating visual information through VLMs. Text-based approaches often generate text labels or bounding boxes for each visual element to guide agent actions [ 46 , 22 ] . However, they can struggle with visually entangled tasks where textual descriptions are ambiguous, incomplete, or fail to capture crucial visual features (e.g., floating windows), even when using accessibility trees. On the other hand, vision-based pipelines [ 11 , 28 ] often rely heavily on the VLM’s ability to ground visual elements. We observe that many errors arise from inadvertently clicking empty or incorrect interface components, underscoring the limitations of existing single-inference visual grounding methods using VLMs. Once an error occurs, there is no feedback loop to correct it, causing mistakes to compound throughout the process. Figure 1 : Overview . When GUI agents encounter execution errors, we instruct the model to focus on a specific point of interest and extract multiple sub-regions around this focal point (1). The agent then independently generates candidate actions for each sub-region. Actions that interact with a specific coordinate are marked with pink-star landmarks (e.g., “click”) to visually indicate the relevant location (2). We retain the pink-star landmarks to track interaction history of RegionFocus for diverse exploration (3). Motivated by these shortcomings, we propose a visual test-time scaling framework, RegionFocus, designed to narrow the GUI model’s attention to salient interface regions when execution errors occur or other conditions are triggered (e.g., VLM self-judgment). Specifically, as illustrated in Figure 1 (1), we leverage the VLM’s capability to identify points of interest and combine this with bounding-box proposals generated either from fixed-ratio masks or segmentation models such as SAM [ 16 ] . For each sub-region, the agent independently predicts actions based solely on the local context (Figure 1 (2)), subsequently aggregating the top candidate actions to form a refined, single-step response. Furthermore, interactions with web or OS interfaces allow our method to zoom into targeted areas, enhancing the resolution of selected regions for more careful examination. RegionFocus works as a modular plug-in for GUI agents without affecting the original workflow. In order to keep track of regions visited by RegionFocus, we introduce an image-as-map mechanism to record temporal information. In this approach, elements previously considered by the agent are annotated in the UI screenshot with visual landmarks (e.g., the pink-stars in Figure 1 (3)). These landmarks prevent the agent from revisiting regions it has already examined and guide the agent towards unexplored areas. These markers do not interfere with the model’s regular inference process which leverages unaltered webpage screenshots, and are only used in the RegionFocus component. Once the agent navigates to a new page, all landmarks are refreshed to have a new RegionFocus history. In addition to representing RegionFocus history, we also leverage image-as-map in the action aggregation process (Figure 1 (2)), where candidate actions are represented using landmarks for selecting an optimal action. We find image-as-map to be highly effective in representing both temporal information (e.g., previously visited regions) and spatial information (e.g., multiple action candidates) compared to alternative representations such as element coordinates represented in the form of text alone. This is particularly crucial for distinguishing between screen elements in close proximity, which is challenging to reason about based on a text representation of the coordinates alone. With our proposed visual test-time scaling framework, we help existing models—such as UI-TARS [ 28 ] and QWen2.5-VL [ 3 ] —achieve better performance in both web-based and desktop interfaces. In particular, we demonstrate substantial performance gains on benchmarks including ScreenSpot-Pro [ 18 ] for OS-level GUI navigation, as well as WebVoyager [ 13 ] for browser automation. Through our experiments, we show that even a simple fixed-ratio bounding-box generation approach yields pronounced improvements over baseline systems, underscoring the efficacy of focusing the model’s attention on visually relevant regions. Our empirical studies further indicate that image-as-map consistently outperforms text-based representations for VLM agents. Overall, our findings highlight the value of visual test-time scaling as a simple yet powerful extension to existing VLM-based GUI agents. Figure 2 : Overview of integrating RegionFocus into GUI agent pipelines. The standard inference process (blue arrow) takes input information and continuously predicts subsequent actions. When the GUI agent encounters errors, RegionFocus (green arrow) activates, proposing focal points to extract targeted sub-regions. Actions are then predicted individually for these sub-regions and aggregated into a single refined action for standard inference. 2 Related Work 2.1 GUI Agents Recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs) have significantly enhanced GUI automation, enabling agents to effectively interact with diverse graphical environments through textual and visual modalities [ 40 , 8 , 38 , 15 , 34 , 11 , 21 , 33 , 27 , 31 , 24 ] . Prior studies generally adopt two distinct approaches: (1) text-based reasoning, which leverages structured representations such as HTML or accessibility trees [ 17 , 5 , 47 ] , extracting structured interface information [ 22 , 43 ] and supplementary textual details for input into LLMs/VLMs [ 46 ] ; and (2) vision-based inference, relying on VLMs to