BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent

Shaojie Zhang Ruoceng Zhang Pei Fu Shaokang Wang Jiahui Yang Xin Du Shiqi Cui Bin Qin Ying Huang Zhenbo Luo Jian Luan MiLM Plus, Xiaomi Inc{zhangshaojie5, zhangruoceng1, fupei1, luozhenbo, luanjian}@xiaomi.com · 2025

Read BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent by Shaojie Zhang Ruoceng Zhang Pei Fu Shaokang Wang Jiahui Yang Xin Du Shiqi Cui Bin Qin Ying Huang Zhenbo Luo Jian Luan MiLM Plus, Xiaomi Inc{zhangshaojie5, zhangruoceng1, fupei1, luozhenbo, luanjian}@xiaomi.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15566.txt.

Introduction

Shaojie Zhang leads a large team that makes screen using agents act more like people. Current agents that click through apps reason in ways unlike human use. The team split the act into three phases named Blink, Think, and Link. Blink spots key screen areas fast, like an eye jump. Think plans at a higher level, and Link turns the plan into commands.

They add two tools to the frame. One builds training data for the Blink phase on its own. The other is a rule based reward that scores both the process and the outcome, a first of its kind for this frame. Their model builds on this setup for human style screen control.

Catalog record

CREATORShaojie Zhang Ruoceng Zhang Pei Fu Shaokang Wang Jiahui Yang Xin Du Shiqi Cui Bin Qin Ying Huang Zhenbo Luo Jian Luan MiLM Plus, Xiaomi Inc{zhangshaojie5, zhangruoceng1, fupei1, luozhenbo, luanjian}@xiaomi.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent Shaojie Zhang Ruoceng Zhang Pei Fu Shaokang Wang Jiahui Yang Xin Du Shiqi Cui Bin Qin Ying Huang Zhenbo Luo Jian Luan MiLM Plus, Xiaomi Inc{zhangshaojie5, zhangruoceng1, fupei1, luozhenbo, luanjian}@xiaomi.com Abstract In the field of AI-driven human-GUI interaction automation, while rapid advances in multimodal large language models and reinforcement fine-tuning techniques have yielded remarkable progress, a fundamental challenge persists: their interaction logic significantly deviates from natural human-GUI communication patterns. To fill this gap, we propose “Blink-Think-Link” (BTL), a brain-inspired framework for human-GUI interaction that mimics the human cognitive process between users and graphical interfaces. The system decomposes interactions into three biologically plausible phases: (1) Blink - rapid detection and attention to relevant screen areas, analogous to saccadic eye movements; (2) Think - higher-level reasoning and decision-making, mirroring cognitive planning; and (3) Link - generation of executable commands for precise motor control, emulating human action selection mechanisms. Additionally, we introduce two key technical innovations for BTL framework: (1) Blink Data Generation - an automated annotation pipeline specifically optimized for blink data, and (2) BTL Reward – the first rule-based reward mechanism that enables reinforcement learning driven by both process and outcome. Building upon this framework, we develop a GUI agent model named BTL-UI, which demonstrates consistent state-of-the-art performance across both static GUI understanding and dynamic interaction tasks in comprehensive benchmarks. These results provide conclusive empirical validation of the framework’s efficacy in developing advanced GUI Agents. † † ∗ Equal contribution; \dagger Corresponding author. † † § https://github.com/xiaomi-research/btl-ui 1 Introduction The automation of graphical user interface (GUI) interactions constitutes a pivotal milestone in developing genuinely intelligent digital assistants wang2024gui ; hu2024agents ; nguyen2024gui . Recent breakthroughs in large vision-language models (VLMs) wang2024qwen2 ; bai2025qwen2 and reinforcement learning fine-tuning techniques have substantially enhanced agents’ capabilities in natural language command interpretation, visual element perception, and multi-step task execution through human-like reasoning hong2024cogagent ; cheng2024seeclick . However, current mainstream systems primarily adopt two approaches. The first relies on supervised fine-tuning (SFT) to align model behavior with task objectives, but this method faces two major limitations: a strong dependence on large-scale expert-labeled data and limited generalization capability when faced with out-of-distribution scenarios. The second approach involves rule-based reinforcement fine-tuning (RFT) guo2025deepseek , as shown in Figure 1 (a), which enhances generalization in complex tasks by using a structured reasoning format: intermediate cognitive steps are encapsulated within <think> tags, and final decisions are expressed through <answer> tags. While effective in improving task performance, these methods qin2025ui ; xia2025gui ; liu2025infigui exhibit two critical shortcomings: (1) significant deviation from natural human-GUI interaction patterns, and (2) excessive focus on interaction outcomes while lacking effective process-oriented reward mechanisms. Figure 1 : Framework comparison of previous “Think-Answer” and “Blink-Think-Link” in GUI tasks for RFT. Specifically, colorful text is supervised by rule-based reinforcement learning. And different colors of text indicate different reward rules. The previous “Think-Answer” framework is optimized by format reward, action type reward, and corresponding args reward. And our “Blink-Think-Link” framework is optimized by dual format reward, blink reward, and link reward. Cognitive studies liversedge2000saccadic ; jaimes2007multimodal ; jacob1991use demonstrate that human-GUI interaction achieves remarkable efficiency through three sequential processes: (a) Blink Phase . Rapid target location during saccadic intervals; (b) Think Phase . Multimodal information integration with intentional reasoning; (c) Link Phase . Generation of precise motor execution commands. Building upon this cognitive finding, we innovatively propose a biologically inspired interaction paradigm—the B link- T hink- L ink ( BTL ) paradigm—for GUI agents, and computationally simulate this paradigm through a structured output mechanism (as shown in Figure 1 (b)): • <blink> : Where relevant areas of the screen are rapidly located, analogous to saccadic eye movements. The visual attention-related region-of-interest information is encapsulated within <blink></blink> tags. • <think> : Where the system engages in high-level reasoning and decision-making, mirroring cognitive task planning. The reasoning processes are recorded in <think></think> tags. • <link> : Where actionable commands are generated for precise execution, reflecting human action selection mechanisms. The action commands are output in <link></link> tags. Specifically, to model human visual localization capabilities during blink intervals, we developed an innovative blink data generation pipeline to automatically produce several region-of-interest (ROI) annotations for training samples. Furthermore, to address the limitations of current reward models in rule-based RFT algorithms that over-rely on outcome-based rewards while neglecting guidance for intermediate interaction processes, We propose the innovative BTL Reward , a Process-Outcome Integrated Reward Mechanism, which comprises three core components: (1) the Dual Format Reward for template and content matching, (2) the Blink Reward for fine-grained guidance of interaction processes, and (3) the Link Reward for action outcome evaluation. By combining the Blink Reward’s granular process supervision with the Link Reward’s precise outcome feedback, this mechanism pioneers the organic integration of process-oriented and outcome-driven approaches. Compared to conventional reward schemes focusing solely on final outcomes, the BTL reward mechanism delivers more sophisticated and multi-dimensional training guidance. Finally, building upon this framework, we develop BTL-UI, a GUI Agent that demonstrates the framework’s effectiveness across multiple GUI tasks. Overall, the main contributions are summarized as follows: 1. We propose BTL (Blink-Think-Link), an innovative framework that simulates the human cognitive process in human-GUI interaction by explicitly modeling how users perceive, process, and act upon interface elements. 2. We propose two key innovations to jointly advance the learning of GUI agents within this framework: (1) Blink Data Generation—an efficient data annotation pipeline that automatically generates multi-region Regions of Interest (ROIs) for training samples; (2) BTL Reward—the first rule-based Process-Outcome Integrated Reward Mechanism. 3. We develop BTL-UI, a GUI agent trained via BTL framework, extensive experiment results demonstrate that the model achieves state-of-the-art performance across multiple GUI benchmarks. Figure 2 : Overall framework of BTL. We adopt GRPO to optimize the proposed BTL. Firstly, the base model generates N completions for a given GUI task sample. Furthermore, GRPO computes the relative advantages within a group of completions, eliminating the need for manually annotated data. Finally, the policy model updates parameters under the guidance of relative advantages and the KL divergence constraint. 2 Related Work 2.1 GUI Agents Autonomous agents powered by LLMs and VLMs have recently garnered considerable scholarly interest due to their interactive functionalities. For GUI tasks, earlier systems relied on LLMs to read and interpret structured representations such as HTML and accessibility trees. However, since icons, images, diagrams, and their spatial relationships are difficult to express in such structured languages, LLM-based agents often perform poorly hong2024cogagent ; nong2024mobileflow ; song2024visiontasker . Therefore, VLM-based agents are introduced to perceive visual GUI signals directly with better performance hu2024agents ; liu2024autoglm ; shen2024falcon ; tang2025think ; christianos2024lightweight ; zheng2025vem . For instance, UGround gou2024navigating developed a specialized GUI grounding model for GUI element localization. OS-Atlas wu2024atlas proposed a foundational model for GUI agents by interpreting human intentions and predicting actions in the form of function calls. Aguvis xu2024aguvis integrated explicit planning and reasoning within the model, enhancing its ability to navigate and interact with complex digital environments autonomously. UI-TARS qin2025ui combines GUI-related pretraining with task-level reasoning fine-tuning to better capture the complexity of GUI interactions. Although research on VLM-based GUI agents has made impressive progress, they mainly follow the training paradigm of SFT,

Read the complete text