S h o w UI- π \pi : Flow-based Generative Models as GUI Dexterous Hands
Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou · 2025
Read S h o w UI- π \pi : Flow-based Generative Models as GUI Dexterous Hands by Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24965.txt.
Introduction
Siyuan Hu, Kevin Qinghong Lin, and Mike Zheng Shou build ShowUI-pi, a flow-based generative model that acts as a dexterous hand inside graphical user interfaces. Existing interface agents predict discrete click points, which blocks free-form closed-loop actions like dragging a slider that need continuous perception and adjustment. Their design unifies discrete clicks and continuous drags in one shared model.
Flow-based generation produces smooth trajectories rather than single points, enabling on-the-fly correction during a drag. AI agent readers get a step toward human-like digital manipulation. Many real tasks, from drawing to data entry with sliders, require exactly that continuous control rather than isolated clicks.
Catalog record
| CREATOR | Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24965
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
S h o w UI- π \pi : Flow-based Generative Models as GUI Dexterous Hands Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou Abstract Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions ( x , y ) (x,y) , which prohibits free-form, closed-loop trajectories ( e.g., dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI- π \pi , the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete–Continuous Actions , integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark , where we manually collect and synthesize 20K drag trajectories across five domains ( e.g., PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents’ drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag ( e.g., Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI- π \pi achieves 26.98 with only 450M parameters , underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi . 1 Introduction Building intelligent assistants capable of dexterous manipulation is essential for achieving human-like automation in both physical and digital environments 21 ; 18 ; 44 . For example, in the physical world, robotic dexterous hands are deployed for manipulation tasks e.g., , sorting objects on physical desktops 4 ; 52 ; 3 ; 30 . Analogously, in the digital world, Graphical User Interfaces (GUI) serve as the primary medium through which people interact with computers, and automating GUI interactions holds substantial promise for enhancing productivity 11 , reducing workload 45 , and improving accessibility 10 . Recent advances in large language models (LLMs) 27 ; 1 ; 16 ; 50 ; 12 and vision-language models (VLMs) 36 ; 37 ; 35 ; 15 have accelerated progress in GUI agents 21 ; 6 . These emerging visual-centric agents directly perceive screen observations and output actions such as clicking or typing. However, most existing GUI agents 42 ; 38 are obtained by fine-tuning foundation VLMs without architecture adaptation, which represent action’s coordinate in a discrete, tokenized form. This representation fundamentally restricts agents from executing complex high-degree-of-freedom dragging, such as creative drawing or Captcha-solving by rotation i.e . , tasks inherently demanding continuous, real-time visual observation and responsible for incremental trajectory adjustment, By contrast, Vision-Language-Action models 9 ; 3 ; 30 in robotics leverage flow-based generative methods ( e.g., diffusion policy, flow matching) to enable achieving such continuous, fine-grained control on the fly. Motivated by how humans control the mouse for fine-grained cursor movement, i.e . , continuously perceive and adjust actions, we wonder: can we construct such a digital dexterous hand in GUI? We propose ShowUI- \pi , the first flow-based GUI model designed for continuous trajectory. Specifically, ShowUI- \pi highlights the following architecture: (i) Unified Discrete-Continuous Actions: ShowUI- \pi casts discrete clicks as drags with negligible movements, and integrates them with continuous drags into a unified modeling. Under this formulation, both action types are represented by a sequence of (x,y,m) triplets, where (x,y) are cursor coordinates and m\in\{\texttt{down},\texttt{up}\} is the mouse button state. This unified design allows ShowUI- \pi to handle both drag and click tasks with a single shared model, adapting without task-specific head selection. (ii) Flow-based Action Generation: Different from existing GUI agents predicting discrete, tokenized actions from language decoding, e.g., , click(x,y) and drag(start, end) , ShowUI- \pi is a flow-based generative model, and employs a lightweight action expert to incrementally predict cursor adjustments from continuous visual observations. Built on a transformer backbone, the action expert is trained with flow matching to generate stable and precise action trajectories; (iii) ScreenDrag benchmark: We construct a benchmark specially for continuous GUI tasks, including 505 real-world drag tasks across five domains covering both professional control and daily usage, such as PowerPoint, OS Desktop and file manager, Handwriting on canvas, Adobe Premiere Pro, and Captcha solving, with 101 tasks for each domain. Moreover, to fully evaluate continuous trajectories, we introduce two complementary modes: an offline open-loop evaluation using average trajectory error and endpoint accuracy, and an online closed-loop evaluation using task success rate. (iv) Continuous Trajectory Training Data: To advance the model training, we construct a training dataset of 20K manually collected and synthesized drag trajectories across the five aforementioned domains and 11 categories of tasks, and all trajectories have recorded UI states and dense coordinates. Our experiments show that proprietary GUI agents still struggle on ScreenDrag ( e.g., Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI- \pi achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. Moreover, our ablation studies further reveal the superficial nature of standard flow-matching training and highlight the substantial impact of individual designs. Our contributions are three-fold: • The first flow-based GUI agent for continuous trajectories. To the best of our knowledge, we are the first work to tackle continuous trajectory-based drags in GUI automation, revealing core limitations of discrete, tokenized actions. We propose ShowUI- \pi , a lightweight 450M flow-based VLA that unifies discrete clicks and continuous drags within a shared modeling. • ScreenDrag training dataset and benchmark suite. We introduce ScreenDrag, a benchmark tailored for continuous GUI manipulation, with 505 real-world drag tasks across five domains and 20K manually collected and synthesized drag trajectories over 11 task categories. ScreenDrag supports both offline open-loop metrics and online closed-loop success evaluation. • Comprehensive evaluation and key insights. Experiments show that proprietary GUI agents still struggle on ScreenDrag ( e.g., , Operator 13.27, best Gemini-2.5-CUA 22.18), while ShowUI- \pi reaches 26.98 with only 450M parameters, highlighting both task difficulty and model effectiveness. Ablations further demonstrate the impact of our individual design choices for achieving robust continuous control. 2 Related Work Figure 2 : ScreenDrag Automated Data Collection Pipeline. ScreenDrag automated data generation pipeline for continuous trajectory-based GUI interaction data. The pipeline includes three stages: (i) Element Parsing: The software application UI is parsed with UI Automation of Windows SDK in order to retrieve the UI element metadata. (ii) Task Proposal: Given the UI element metadata, an LLM will be prompted to generate a drag instruction, the expected metadata change and the drag code with dense trajectory. (iii) Trajectory Synthesis: The drag code will be executed in the software environment. A rule-based verifier will check the parsed metadata from UI states before and after the drag to ensure that the metadata change satisfies the expectation. 2.1 Digital GUI Automation. LLM-based agents in digital environments have evolved from reasoning and tool-use paradigms, such as Chain-of-Thought 39 , ReAct 46 , and related prompting frameworks 47 ; 43 ; 8 , toward task-oriented GUI automation. Existing methods include