DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

Yichao Liu, Huawen Shen, Shiyu Liu, Zeyu Chen, Yu Zhou · 2026

Read DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding by Yichao Liu, Huawen Shen, Shiyu Liu, Zeyu Chen, Yu Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15542.txt.

Introduction

Yichao Liu, Huawen Shen, Liu Yu, Shiyu Liu, Zeyu Chen, and Yu Zhou present DRS-GUI, a no-train frame that helps multimodal models find screen items. Busy high-resolution screens hide the right items amid clutter. A light perceptor probes with three human-like moves: focus, shift, and scatter. A tree-search planner times those moves. A quality score then picks the most apt region. It lifts grounding on ScreenSpot-Pro by 14 percent for both broad and GUI-tuned models.

Catalog record

CREATORYichao Liu, Huawen Shen, Shiyu Liu, Zeyu Chen, Yu Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35365
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15542

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding Yichao Liu, Huawen Shen, Shiyu Liu, Zeyu Chen, Yu Zhou Abstract GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions. However, accurately grounding instruction-relevant elements from high-resolution screenshots cluttered with irrelevant UI components remains challenging for existing approaches. Inspired by how humans dynamically adjust their perceptual scope to locate task-related regions on complex screens, we propose DRS-GUI, a training-free dynamic region search framework for GUI grounding that can be seamlessly integrated into existing MLLMs. DRS-GUI introduces a lightweight UI Perceptor that performs three human-like perceptual actions (Focus, Shift, and Scatter) to progressively explore the interface and generate region proposals. To dynamically schedule these actions, we further design an Action Planner based on Monte Carlo Tree Search (MCTS). A region quality reward is employed to evaluate and select the highly instruction-relevant region, efficiently pruning redundant UI elements. Experiments demonstrate that DRS-GUI yields a 14% improvement on ScreenSpot-Pro for general and GUI-specific MLLMs (Qwen2.5-VL-7B and UGround-V1-7B), significantly enhancing grounding performance and generalization. 1 Introduction Figure 1 : (a) Single-stage methods progressively crop and zoom through forward-only focus, causing error accumulation. (b) DRS-GUI uses an action planner and a UI Perceptor to explore regions via three perceptual actions, guided by region quality evaluation to select the instruction-relevant region. Graphical user interface (GUI) agents, powered by recent advances in multimodal large language models (MLLMs) 43 ; 11 ; 46 ; 42 ; 27 , are transitioning from rule-based scripted control to natural language interaction. The reliability of such interaction fundamentally depends on grounding -accurately localizing the executable UI element specified by a user instruction 20 ; 50 . However, real GUI screenshots are often high-resolution and visually dense, with substantial redundant elements that distract MLLMs from the instruction-relevant cues needed to reliably isolate the true target 19 . Resulting failures arise not only from visual clutter, but also from the absence of an explicit mechanism for adapting perceptual scope during interaction. Existing GUI grounding approaches typically fall into two paradigms, yet both lack truly adaptive perceptual behavior. The first paradigm formulates grounding as a single-step full-screen prediction, directly regressing a point or bounding box over the entire screenshot 17 ; 9 ; 41 . Without explicit control over attention, such models often fail to isolate instruction-relevant cues from background clutter. The second paradigm adopts multi-step refinement strategies, such as iterative cropping or zooming 16 ; 23 ; 36 , to gradually narrow the view toward the target. Although these methods provide progressive focus, their refinement processes are typically forward-only and lack mechanisms to assess whether each step improves alignment with the instruction or to backtrack when early decisions deviate from the correct trajectory 39 . As illustrated in Fig. 1 (a), when the trajectory drifts away from semantic cues, the final region may fail to contain the target, making the error irrecoverable. In contrast, human behavior follows a more flexible pattern: rather than remaining on a single narrowing path, humans scan the layout, refine attention gradually, and step back or shift attention to alternative regions when uncertainty arises. This adaptive and reversible search process naturally evaluates whether attention is moving toward or away from the true target 31 ; 38 ; 37 , a capability that current GUI grounding pipelines lack. These observations suggest that a reliable GUI grounding framework should exhibit two key capabilities: (1) dynamic perception , the ability to revise or shift the viewing region when current evidence is weak rather than committing to a single irreversible narrowing path; and (2) region quality evaluation , where each perceptual step provides a measurable signal indicating whether the search is moving closer to the target, helping to prevent early errors from compounding. To realize these capabilities, we propose DRS-GUI , a training-free framework that introduces a dynamic region search and decision procedure to guide the grounding process. DRS-GUI dynamically adjusts the perceptual scope by proposing candidate regions, evaluating their relevance to the instruction through semantic and structural cues, and selecting refinement or redirection actions when necessary. This adaptive perceptual process echoes how humans selectively explore and adjust attention within visually dense interfaces. To support this process, DRS-GUI incorporates an auxiliary UI Perceptor , built upon a UI element parser 21 and a text-embedding module 29 . The Perceptor provides structured UI representations that help determine which regions carry instruction-relevant cues. Guided by these cues, the system executes three perceptual actions: Focus (contract the view for finer detail), Shift (relocate attention to alternative regions), and Scatter (expand outward to recover broader context). To dynamically schedule these actions, DRS-GUI employs an Action Planner that integrates Monte Carlo Tree Search (MCTS) 28 to enable human-like planning of field-of-view adjustments. We further design a reward that evaluates region quality and determines whether a candidate region should be further explored or selected as final. This step-wise evaluation mitigates the accumulation of early errors, addressing the inherent brittleness of purely forward refinement. DRS-GUI requires no additional model training or fine-tuning, making it a plug-and-play enhancement for existing MLLMs 4 ; 8 . By dynamically generating and evaluating perceptual regions, DRS-GUI reduces redundant visual clutter and improves grounding robustness on high-resolution, element-dense interfaces. In summary, our contributions are as follows: • We propose DRS-GUI, a training-free GUI grounding approach that performs dynamic region search, effectively enhancing MLLMs grounding performance. • We introduce a human-like perception–action process to obtain a reliable target region: UI Perceptor provides instruction-relevant cues to drive three perceptual actions, and an MCTS-based Action Planner with a region quality reward regulates these actions to reach a stable step-wise decision. • Experiments show that DRS-GUI achieves significant improvements on the high-resolution ScreenSpot-Pro benchmark, highlighting its robustness and generality. 2 Related Work GUI Agents. The capability of GUI agents to automate complex tasks 10 ; 44 ; 33 has surged with the advent of powerful Multimodal Large Language Models (MLLMs) 2 ; 32 ; 8 ; 47 . A cornerstone of this capability is grounding 41 ; 52 . Current approaches mainly fall into two paradigms. The first is a static full-screen paradigm, where representative systems such as SeeClick 9 , OS-Atlas 42 , and UGround 11 construct large-scale annotated datasets to fine-tune MLLMs for text-to-coordinate prediction. While effective in structured layouts, this static paradigm often degrades on high-resolution, element-dense interfaces, where visual clutter disperses attention and leads to unstable localization. The limitations of static models, together with the recent rise of “think-with-image” strategies 54 ; 53 ; 6 , have motivated a second paradigm based on iterative refinement. These methods employ a multi-step zoom-in process: ECP 16 and R-VLM 23 first localize candidate regions before high-resolution refinement, DiMo-GUI 39 performs training-free GUI grounding via modality separation and iterative region zoom-in, and LASER 36 is a self-evolving framework that equips VLMs with multi-step perception and adaptive region reasoning for precise instruction-relevant coordinate prediction. However, these zoom-in approaches are inherently irreversible: once the correct region is discarded early, the search cannot recover and errors accumulate. By contrast, our method employs a dynamic perception planner that can redirect and revisit regions as needed, yielding more reliable grounding in dense interfaces. Visual Search. Early visual

Read the complete text