UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen · 2026

Read UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding by Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14113.txt.

Introduction

Tang and colleagues make GUI grounding zoom adaptively instead of uniformly. UI-Zoomer treats both the zoom trigger and the zoom scale as uncertainty problems: a confidence gate blends agreement across random candidates with token-level confidence, firing only when localization is uncertain. An uncertainty-driven crop module then sizes each crop from prediction variance. Gains reach up to 13.4 percent across ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, with no extra training.

Catalog record

CREATORFei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43040
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen Abstract GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose UI-Zoomer , a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4%, +10.3%, and +4.2% respectively, with no additional training required. † † affiliation: Zhejiang University † † affiliation: Ant Group † † contribution: Equal Contribution † † contribution: Corresponding authors † † date: August 10, 2026 † † Project Page: https://zju-real.github.io/UI-Zoomer † † Code: https://github.com/ZJU-REAL/UI-Zoomer † † correspondence: [email protected] 1 Introduction Grounding natural language instructions to interface elements is a fundamental capability for autonomous GUI agents Gou et al. 2024 ; Tang et al. 2025a ; Tang et al. 2025c ; Xu et al. 2025 ; Hong et al. 2024 ; Lin et al. 2024 ; Jiang et al. 2025 ; Yang et al. 2023 . Despite significant progress through supervised fine-tuning and reinforcement learning Qin et al. 2025 ; Xu et al. 2024 ; Xie et al. 2025 ; Yuan et al. 2025 ; Gu et al. 2025 , models still fail systematically on small icons and dense layouts in complex interfaces Li et al. 2025 . A natural remedy is test-time zoom-in scaling : crop a region of the screenshot and re-run the model at higher effective resolution Wu et al. 2025a ; Luo et al. 2025 ; Nguyen 2024 ; Lee et al. 2025 .While this paradigm has shown clear promise for fine-grained GUI localization Wu et al. 2025a ; Luo et al. 2025 , a more fundamental question remains unaddressed: which instances actually need zoom-in, and how much should we zoom? Existing zoom-in methods share two fundamental limitations. First, they apply cropping indiscriminately: Wu et al. 2025a zooms in unconditionally on every sample with a fixed scaling factor, while Luo et al. 2025 triggers zoom-in only upon execution errors, with no regard to whether the model is actually uncertain on the instance at hand. We show empirically that unconditional zoom-in on ScreenSpot-v2 degrades accuracy below the direct prediction baseline while significantly increasing latency (Table 1 ), as easy cases lose the global context the model was already exploiting. Second, all existing methods fix the crop window to a predetermined ratio Wu et al. 2025a ; Luo et al. 2025 ; Lee et al. 2025 , regardless of whether candidates are tightly clustered or widely scattered, leaving the crop either too broad to improve resolution or too narrow to retain critical context. Figure 1 : Comparison of GUI grounding paradigms. (a) Direct grounding methods struggle with dense interfaces. (b) Iterative cropping methods incur large resource costs and use rigid cropping ratios. (c) Our UI-Zoomer applies Test-Time Scaling (TTS) with reliability gating, adaptively choosing between consensus voting and adaptive cropping, achieving stronger robustness with one-take time costs. Method Avg Acc Time w/o DiMo-GUI 81.84% 35:47 w/ DiMo-GUI 77.20% 6:43:07 Table 1: Accuracy and inference time of w/ and w/o iterative cropping on ScreenSpot-V2. The root cause is that these methods treat all instances uniformly, without consulting the model’s own prediction behavior. Recent work shows that spatial agreement across stochastic samples correlates with localization reliability Du et al. 2025 , and that coordinate likelihoods near a predicted point follow a smooth Gaussian distribution in pixel space Lee et al. 2025 , confirming that VLMs implicitly encode continuous spatial uncertainty. The variance of sampled predictions Wang et al. 2026 ; Du et al. 2025 thus encodes both whether the model is confused and over what spatial extent, which is precisely the information needed to gate zoom-in and size the crop window. This motivates a simple but previously unexplored principle: zoom only when uncertain, and zoom by how much the predictions disagree . Building on this insights, we propose UI-Zoomer , a training-free adaptive zoom-in framework for GUI grounding. UI-Zoomer first draws N stochastic candidates from the model and computes a reliability score by fusing spatial consensus with token-level confidence; instances that pass the gate are resolved immediately by consensus voting. For uncertain instances, the crop window is derived from the variance of candidate predictions decomposed into inter-sample positional spread and intra-sample box extent, yielding a per-instance radius that contracts for easy cases and expands for hard ones. A single deterministic re-inference pass on the resulting crop completes the refinement. Extensive experiments on three widely-adopted GUI grounding benchmarks, ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, demonstrate that UI-Zoomer consistently improves over strong baselines, achieving gains of up to +13.4%, +10.3%, and +4.2% respectively. Icon targets benefit more than text targets on average, consistent with the intuition that compact and semantically ambiguous elements profit most from high-resolution refinement. Ablations confirm the independent contribution of each component and the advantage of adaptive crop sizing over any fixed-ratio alternative. Our contributions are threefold: • We propose UI-Zoomer, a training-free adaptive zoom-in framework that frames the trigger and scale of zoom-in as a prediction uncertainty quantification problem. • UI-Zoomer comprises a confidence-aware gate that avoids unnecessary computation by routing only uncertain instances to refinement, and a Gaussian-based adaptive crop sizing module that derives the crop window from the variance of candidate predictions. • Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements across four model architectures, with gains of up to +13.4% on ScreenSpot-Pro. 2 Related Work 2.1 GUI Grounding GUI grounding requires predicting the pixel coordinates of an interface element given a screenshot and a natural language instruction. Early work builds pipeline-based systems that chain OCR, icon detectors, and LLMs for planning and element selection Zhang et al. 2025 ; Wang et al. 2024 ; Li et al. 2024 ; Agashe et al. 2024 ; Zhang et al. 2025 ; Liu et al. 2024 ; Yang et al. 2023 ; Bai et al. 2021 ; Xu et al. 2024 ; Wu et al. 2024 ; Tang et al. 2025b ; Wu et al. 2025b ; Agashe et al. 2025 . A second generation trains specialist VLMs end-to-end on large-scale GUI corpora, with models such as UGround Gou et al. 2024 , OS-Atlas Wu et al. 2024 , and UI-TARS Qin et al. 2025 demonstrating strong cross-platform generalization. More recently, reinforcement fine-tuning has emerged as a data-efficient alternative: methods including UI-R1 Lu et al. 2025a , GUI-G 2 Tang et al. 2025a , SE-GUI Yuan et al. 2025 , and UI-Venus Gu et al. 2025 apply

Read the complete text