GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu, S-Lab, Nanyang Technological University, SenseTime ResearchProject Page: https://penghao-wu.github.io/GUI_Reflection/ · 2025

Read GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior by Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu, S-Lab, Nanyang Technological University, SenseTime ResearchProject Page: https://penghao-wu.github.io/GUI_Reflection/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08012.txt.

Introduction

Penghao Wu and five co-authors teach interface agents to reflect on and fix their own errors, in a paper from 2025. Most training data for screen control is nearly error-free, so models never learn to recover when a tap goes wrong. Real use demands spotting and undoing such slips.

Their framework adds reflection training in three stages, from screen-focused pre-training to online tuning on real devices. Data pipelines build reflection examples from successful runs with no human labels. A task suite tests reflection skills directly, and an online loop keeps sharpening recovery in a live phone setting. Models gain self-correction without hand-built data.

Catalog record

CREATORPenghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu, S-Lab, Nanyang Technological University, SenseTime ResearchProject Page: https://penghao-wu.github.io/GUI_Reflection/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu, S-Lab, Nanyang Technological University, SenseTime ResearchProject Page: https://penghao-wu.github.io/GUI_Reflection/ Abstract Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection , a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning . GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly. Figure 1 : Illustrative comparison of typical GUI models versus our proposed GUI model with self-reflection behaviors. While current models fail to recognize and recover from errors (left), our model (right) demonstrates the ability to: 1. Recognize its mistake; 2. Undo the incorrect action and get back on track; 3. Summarize the mistake and make another try, ultimately succeeding. 1 Introduction Graphical User Interface (GUI) automation stands as a critical frontier for enhancing productivity and accessibility across the vast landscape of digital applications and devices. The advent of large language models (LLMs) and multimodal large language models (MLLMs) has catalyzed significant progress in this area. Typical GUI agents can be mainly categorized into two groups: agent-based frameworks and end-to-end GUI models. Agent-based frameworks [ 49 , 52 , 40 , 39 , 1 ] usually leverage the reasoning and generalization capabilities of some foundation models ( e.g. GPT-4o [ 20 ] ) with agentic modules and external tools to complete tasks. While representing a leap forward, such agent-based modular frameworks often rely on intricate prompt engineering and complex workflows, suffering from high computation and cascaded errors, potentially limiting their adaptability in real-world scenarios. As for end-to-end multimodal GUI models [ 14 , 43 , 44 , 30 ] which interact with GUIs more like humans do, the perception, reasoning, and action grounding are integrated within a single model. Such models not only promise more adaptable and scalable GUI agents but also provide a potentially valuable avenue for studying broader aspects of artificial general intelligence 1 1 1 See Appendix A for a detailed discussion of related work. . The current paradigm for training end-to-end multimodal GUI models includes a GUI pre-training stage to inject GUI-related knowledge into the base MLLM, followed by a supervised fine-tuning (SFT) stage with demonstration trajectories. However, one big issue of this paradigm is that it relies heavily on offline datasets composed of pre-collected successful interaction trajectories. While this approach teaches models to mimic expert demonstrations for specific tasks, it inherently limits their ability to handle the complexities and unpredictability of real-world interactions. When encountering unfamiliar UI interfaces, incorrect attempts, or execution failures, these models lack the capability to recognize the error, understand its cause, or formulate a corrective plan based on failed attempts. Crucially, even though the base models (the base MLLM before GUI-specific training) they trained on might originally contain certain reflection and reasoning abilities, this offline SFT process, focused solely on successful examples, can inadvertently diminish such capabilities or behaviors. Recent LLM research has shown that through online training like reinforcement learning (RL), the reasoning and reflection abilities of the base models can be greatly enhanced. Moreover, recent studies [ 34 , 48 , 15 ] have shown that the verification and reflection behaviors of the base model are crucial for the success of RL training, and such capabilities in the base models largely affect the performance upper bound in the RL stage. However, for end-to-end multimodal GUI models, the current paradigm makes it difficult to sample or explore potentially corrective or reflective behaviors after the offline SFT stage, and further RL training cannot effectively activate or enhance such reflection abilities. UI-TARS [ 30 ] explores the incorporation of reflection and correction behaviors through an online bootstrapping mechanism. However, its design primarily focuses on the final online stage and depends on human-annotated feedback to guide learning. To address these fundamental limitations, we introduce an automatic framework designed to explicitly integrate self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout different training stages. We first decompose the reflection and error correction ability for GUI agents into three core capabilities: (1) verifying the correctness of previous actions and recognizing errors or deviations, (2) backtracking when deviating from the correct trajectory, and (3) reflecting on erroneous attempts to learn from mistakes and inform subsequent actions. Our framework strategically embeds the learning of these abilities across distinct training phases, including GUI-specific pre-training, offline supervised fine-tuning, and online training, aiming to cultivate GUI models capable of robust error handling and adaptive recovery as illustrated in Fig 1 . Specifically, during the GUI pre-training phase, while current efforts primarily focus on GUI visual grounding and general UI understanding, we identify a critical gap: the lack of explicit training signals for reflection and correction related abilities, which leads to the degradation of reflection behaviors in the base MLLM. To address this, we propose the GUI-Reflection Task Suite with Action Verification, Action Reversal, and Mistake-Informed Reattempt tasks, specifically designed to evaluate and cultivate the reflection-oriented capabilities for GUI models. Besides, we have designed a scalable automatic data pipeline to construct realistic reflection and error correction scenarios derived from existing offline successful trajectories and inject such data into the offline SFT stage. This allows the model to learn the behaviors of reflection and correction. Furthermore, to further improve the GUI models through real interactions, we have built a robust and extensible environment for Android-based tasks. The environment includes 215 programmatic task templates in a distributed framework. Within this learning environment, we design a simple iterative online reflection tuning algorithm. This algorithm allows the model to interact with the environment and receive automatically generated pre-error correction and post-error reflection supervisions to continuously enhance its general capability and reflection and error correction capabilities iteratively. Our main contributions are: 1) We propose GUI-Reflection Task Suite, designed to explicitly train and evaluate the crucial reflection-oriented abilities of GUI models during GUI pre-training. 2) We introduce a scalable, automatic data pipeline to construct reflection and error correction scenarios from existing successful trajectories, enabling the injection of these behaviors during offline SFT without manual annotation. 3) We develop an online learning environment for mobile GUI agents and an iterative online

Read the complete text