Learning Active Perception via Self‑Evolving Preference Optimization for GUI Grounding
Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao Li · 2025
Read Learning Active Perception via Self‑Evolving Preference Optimization for GUI Grounding by Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04243.txt.
Introduction
Wanfu Wang and four co-authors teach vision-language models to zoom in on the right part of a screen in a 2025 research paper. GUI grounding means finding the exact screen element a user's instruction refers to, such as a small button in a crowded window. High-resolution screens make that search hard. Their framework, LASER, trains models to look actively in steps, judging candidate regions by quality scores and overlap with targets, then building preference examples from the best searches.
Training rewards both accuracy and variety, and lets harder screens consume more reasoning steps. On two ScreenSpot benchmarks, scores rose steadily across versions. One tuned model set a new best result among its size class on the professional benchmark.
Catalog record
| CREATOR | Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04243
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Active Perception via Self‑Evolving Preference Optimization for GUI Grounding Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao Li Abstract Vision Language Models (VLMs) have recently achieved significant progress in bridging visual perception and linguistic reasoning. Recently, OpenAI o3 model introduced a zoom-in search strategy that effectively elicits active perception capabilities in VLMs, improving downstream task performance. However, enabling VLMs to reason effectively over appropriate image regions remains a core challenge in GUI grounding, particularly under high-resolution inputs and complex multi-element visual interactions. In this work, we propose LASER , a self-evolving framework that progressively endows VLMs with multi-step perception capabilities, enabling precise coordinate prediction. Specifically, our approach integrate Monte Carlo quality estimation with Intersection-over-Union (IoU)-based region quality evaluation to jointly encourage both accuracy and diversity in constructing high-quality preference data. This combination explicitly guides the model to focus on instruction-relevant key regions while adaptively allocating reasoning steps based on task complexity. Comprehensive experiments on the ScreenSpot Pro and ScreenSpot-v2 benchmarks demonstrate consistent performance gains, validating the effectiveness of our method. Furthermore, when fine-tuned on GTA1-7B, LASER achieves a score of 55.7 on the ScreenSpot-Pro benchmark, establishing a new state-of-the-art (SoTA) among 7B-scale models. Code — https://github.com/wwfnb/Laser 1 Introduction Training autonomous agents to interact with graphical user interfaces (GUIs) presents unique challenges due to the high complexity of visual observations and the compositional nature of user interface tasks ( Xi et al. 2025 ; Qin et al. 2024 ) . However, the effectiveness of such agents can be largely attributed not only to their ability to ground language in visual observations, but also to their capability in active perception. In particular, active perception ( Zhu et al. 2025 ) is critical for guiding attention toward semantically relevant regions, thereby facilitating accurate action selection in complex GUI environments. In GUI grounding tasks, most existing approaches adopt a direct prediction paradigm ( Qin et al. 2025 ; Wu et al. 2024 ; Hong et al. 2024 ; Xu et al. 2024 ) , where the model infers the target location and corresponding action in a single reasoning step. Although widely used, this method often fails in high-resolution or multi-element interaction scenarios, primarily due to the absence of active perception capabilities. Our preliminary analysis shows that the choice of visual focus region plays a pivotal role in the performance of VLMs (Qwen2.5-VL-7B). As illustrated in Figure 1(a) , explicitly focusing on appropriate regions yields substantial performance gains on the ScreenSpot-Pro benchmark ( Li et al. 2025 ) . Such appropriate regions can be characterized as those that (i) minimize background noise and (ii) retain key contextual cues grounded in the user instruction , which are crucial for accurate model prediction. (a) Model performance is affected by varying the size of the “ Focus Region ” around the Ground Truth Bounding Box. (b) The bottom Figure shows the performance curves of our LASER model across different training stages, achieving SoTA performance based on the GTA1-7B backbone. Recent advances such as OpenAI’s o3 ( OpenAI 2025 ) model have introduced a novel paradigm termed “ thinking with image ” ( Su et al. 2025b ) , in which intermediate visual transformations of the input are incorporated into the reasoning process as part of a visual chain-of-thought (CoT). This paradigm enables a new axis of test-time scaling , effectively decomposing complex reasoning tasks into a sequence of simpler, more tractable subtasks. Inspired by the success of long CoT strategies in language reasoning ( Chen et al. 2025 ) , this approach suggests that constructing visual CoT through multi-step reasoning can significantly enhance model performance on challenging visual tasks ( Zhu et al. 2025 ; Zhang et al. 2025 ; Su et al. 2025a ; Zheng et al. 2025 ) . However, how to enable open-source VLMs to acquire such active perception capabilities remains an open and underexplored challenge, particularly in GUI grounding tasks. In this work, we identify two core challenges in enabling active perception for GUI grounding: (1) how to evaluate preferences over candidate focus regions ? and (2) how to allocate the model’s reasoning budget based on task difficulty ? To address the first challenge, we introduce two complementary quality estimation techniques, Monte Carlo and IoU-based estimation, aimed at constructing preference data that jointly emphasizes both accuracy and diversity . The Monte Carlo approach identifies focus regions that consistently lead to incorrect actions and treats them as negative examples, thereby improving the reliability of preference supervision. In contrast, the IoU-based method filters out preference pairs with high spatial overlap to ensure sufficient diversity among candidate regions. To address the second challenge, we construct multi-step reasoning trajectories through an iterative refinement process, where the number of reasoning steps dynamically adapts to the complexity of the task. This mechanism enables the model to allocate its inference budget more effectively. Building on these methods, we further propose a self-evolving optimization framework, named LASER , which enables the model to bootstrap its active perception capabilities through rejection sampling–based SFT and region-wise preference learning, without relying on extensive human supervision. We conduct extensive experiments and analyses of LASER on two high-resolution GUI grounding benchmarks: ScreenSpot-Pro and ScreenSpot-v2. As shown in Figure 1(b) , LASER trained with Qwen2.5-VL-7B achieves a score of 47.5 on ScreenSpot-Pro, surpassing the baseline by +20.7 points and outperforming several RL–based methods, including GUI-R1 and SE-GUI. Furthermore, when trained with GTA1-7B, LASER achieves a score of 55.7, even outperforming the much larger Qwen2.5-VL-72B, demonstrating both the effectiveness and scalability of our method. In summary, our key contributions are as follows: 1. We propose LASER , a self-evolving framework that performs multi-stage region-wise preference optimization, enabling VLMs to progressively focus on crucial regions and adaptively execute multi-step perception. 2. The regions automatically predicted by LASER on the benchmark closely align with the prior distribution shown in Figure 1(a) (approximately 20\% of the original image), which further validates the effectiveness of our region selection strategy. 3. LASER trained with GTA1-7B achieves a new SoTA score of 55.7 on the ScreenSpot-Pro benchmark. 2 Related Work 2.1 Autonomous GUI Agents Training agents to assist humans in accomplishing complex tasks within GUI environments remains a significant challenge. Current mainstream approaches can be categorized into textual-based agents ( Deng et al. 2023 ; Zhou et al. 2023 ; Lai et al. 2024 ) and vision-based agents ( Hong et al. 2024 ; Qin et al. 2025 ; Xu et al. 2024 ) , which rely on analyzing textual representations (such as HTML code or accessibility metadata) and visual inputs (such as interface screenshots), respectively. In many real-world scenarios, textual representations fail to capture the spatial informations and handle complex GUI interface ( Huang et al. 2024 ) . Prior work addresses this challenge through large-scale data synthesis ( Wu et al. 2024 ; Gou et al. 2024 ; Xie et al. 2025 ) , involving collection of human instructions, GUI screenshots, and corresponding grounding coordinates, along with reinforcement learning ( Yang et al. 2025 ; Yuan et al. 2025 ; Lu et al. 2025 ; Luo et al. 2025 ) to encourage deep reasoning capabilities. In contrast, we propose a self-evolving methods that encourages active perception behaviors in VLMs. By progressively