Knowing When to STOP , RECOVER , and SEARCH A Modular Framework for GUI Automation
Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie · 2026
Read Knowing When to STOP , RECOVER , and SEARCH A Modular Framework for GUI Automation by Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21375.txt.
Introduction
Qijun Han and 13 colleagues build VLAA-GUI, a framework teaching GUI agents when to stop, recover, and search. A completeness verifier cross-examines finish claims against visible evidence, a loop breaker switches modes and forces strategy changes after failures, and an on-demand search agent looks up unfamiliar workflows. Coding and grounding agents join for code-heavy and precise actions. Across five backbones on Linux and Windows tasks it leads both benchmarks, scoring 77.5 percent on OSWorld and 61 percent on WindowsAgentArena, with three backbones beating human OSWorld performance.
Catalog record
| CREATOR | Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58974 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowing When to STOP , RECOVER , and SEARCH A Modular Framework for GUI Automation Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie Abstract Autonomous GUI agents face two fundamental challenges: early stopping , where agents prematurely declare success without verifiable evidence, and repetitive loops , where agents cycle through the same failing actions without recovery. We present VLAA-GUI , a modular GUI agentic framework built around three integrated components that guide the system on when to Stop , Recover , and Search . First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step—with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both ( 77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance ( i.e . , 72.4%) on OSWorld in a single pass. In particular, VLAA-GUI with Sonnet 4.6 at only 15 action steps already surpasses the best published 50-step system. Ablation studies show that all three proposed components consistently improve a strong backbone ( e.g . , Sonnet 4.6), while a weaker backbone ( e.g . , Gemini 3 Flash) benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models. 1 Introduction The rapid advancement of multimodal large language models (MLLMs) [ 79 ] has catalyzed a new generation of autonomous Graphical User Interface (GUI) agents [ 49 , 1 , 65 , 66 , 33 ] capable of performing desktop tasks by observing screenshots and executing mouse and keyboard actions. Systems such as OSWorld [ 67 ] , WindowsAgentArena [ 11 ] , and related benchmarks [ 79 , 72 ] have established standardized evaluation environments spanning Linux, Windows, and macOS, revealing both promises and persistent limitations of current approaches. Despite steady progress, two fundamental problems remain largely unsolved. First, agents do not reliably know when a task is finished : they routinely declare success prematurely, e.g . , after opening a “Save As” dialog before writing the file, or after toggling a setting without verifying the state changed—because completion is left to the model’s implicit judgment rather than verified against observable UI evidence [ 3 , 14 ] . Second, agents fall into repetitive loops : cycling through the same failing action without recovery. Moreover, existing anti-looping heuristics operate at a single granularity and cannot escalate across interaction modalities or planning strategies [ 77 , 53 , 44 , 38 , 36 ] . We present VLAA-GUI , a modular GUI agent framework that addresses both challenges through three integrated mechanisms— a Completeness Verifier, a Loop Breaker, and a Search Agent —to regulate the system in knowing when to STOP , RECOVER (from repetitive loops), and SEARCH online for enhanced performance. To address the early stopping issue, we introduce a mandatory Completeness Verifier: a prompt-level Completion Gate that requires the agent to derive UI-observable success criteria and verify them against the current screenshot before every decision, backed by a verifier model that independently cross-examines any completion claim and rejects it under any ambiguity at every finish step. To address repetitive loops, we introduce a mandatory Loop Breaker with three escalation filtering rules: I. switch interaction modality after certain consecutive no-change failures, II. change strategy after a sequence of identical screen states, and III. enforce a mandatory strategy change whenever an external model judge signals a loop. We impose the Loop Breaker after every action step to inform the system when to recover from a potential loop in time. Beyond the two core challenges, GUI agents frequently stall on unfamiliar application workflows, leading to poor out-of-distribution capabilities. Prior work addresses this via visual browser-based search [ 70 ] , which requires additional agent steps and a visual grounding pipeline. We instead deploy an on-demand Search Agent that directly queries a capable LLM with search grounding ( e.g . , Gemini 3 Pro [ 28 ] ), returning plain textual knowledge injected into the Manager’s context, which is faster and more reliable. We further integrate a Coding Agent for programmatic edits and a Grounding Agent for agent action grounding following existing works [ 26 , 70 , 64 ] . We evaluate VLAA-GUI on OSWorld-Verified [ 67 ] and WindowsAgentArena (WAA) [ 11 ] across five top-tier backbones—Claude Opus 4.6 [ 8 ] , Opus 4.5 [ 5 ] , Sonnet 4.6 [ 6 ] , Gemini 3.1 Pro [ 29 ] , and Gemini 3 Flash [ 27 ] —to systematically validate the framework across model families and capability tiers. In Figure 1 , our system achieves 77.5% on OSWorld with Opus 4.6, and three of the five backbones (Opus 4.6 at 77.5%, Opus 4.5 at 74.9%, Gemini 3.1 Pro at 72.5%) surpass human performance ( i.e . 72.4%) in a single pass—making VLAA-GUI the first framework to do so. All five backbones outperform prior SOTA ( e.g . , Agent S3 at 67.5%), and VLAA-GUI w/ Sonnet 4.6 at only 15 action steps already surpasses the best published 50-step system with one-third the budget. On the other hand, VLAA-GUI generalizes well to WAA, reaching 61.0% and outperforming the strongest baselines by over 4%. Beyond the main results, we conduct three lines of ablation analysis. I. We perform controlled incremental ablations on OSWorld and WAA to isolate each component’s marginal contribution, finding that all three components consistently improve a strong backbone ( e.g . , up to +3.1% each with Sonnet 4.6), while a weaker backbone benefits more at more relaxed budgets, since tool use consumes actions that less efficient models cannot afford under tight budgets. II. We analyze the false completion behavior, revealing that over 86% of failures involve the agent incorrectly believing it has succeeded, and that the Completeness Verifier reduces the overall false completion ratio by up to 3.9%. III. We analyze the repetitive-loop behavior, showing that the Loop Breaker Agent nearly halves wasted steps (4.9% \to 2.8%) for loop-prone backbones. Figure 1 : Advantages of VLAA-GUI. (a) Our VLAA-GUI (w/ Opus 4.6) achieves the best results (77.5%) on OSWorld-Verified [ 67 ] and surpasses human performance with one pass. (b) On one hand, by employing Completeness Verifier, VLAA-GUI mitigates the early stopping issue. (c) On the other hand, the proposed Loop Breaker helps ease the repetitive-looping problem and save wasted action steps in GUI tasks. 2 Related Work 2.1 GUI Agent Benchmarks. Standardized benchmarks