Step-GUI Technical Report
GELab-Team, StepFun Homepage : https://opengelab.github.io/ Github : https://github.com/stepfun-ai/gelab-zero Huggingface : Step-GUI Collections · 2025
Read Step-GUI Technical Report by GELab-Team, StepFun Homepage : https://opengelab.github.io/ Github : https://github.com/stepfun-ai/gelab-zero Huggingface : Step-GUI Collections (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15431.txt.
Introduction
Haolong Yan and a large team describe Step-GUI in a 2025 technical report on agents that operate phone and computer screens. Such agents need good training data, but hand labeling screen actions is slow and costly. The report presents a self improving pipeline meant to solve that problem.
Their Calibrated Step Reward System turns model made action paths into trusted training signals, with over 90 percent labeling accuracy at far lower cost. The resulting 4B and 8B models score well on AndroidWorld, OSWorld, and ScreenShot-Pro tests. The team also adds a privacy minded control protocol and a new daily use benchmark, AndroidDaily.
Catalog record
| CREATOR | GELab-Team, StepFun Homepage : https://opengelab.github.io/ Github : https://github.com/stepfun-ai/gelab-zero Huggingface : Step-GUI Collections |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15431
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Step-GUI Technical Report GELab-Team, StepFun Homepage : https://opengelab.github.io/ Github : https://github.com/stepfun-ai/gelab-zero Huggingface : Step-GUI Collections Abstract Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System , which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving > > 90% annotation accuracy with 10~100 × \times lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP , the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily , a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions. \reportnumber 001 Figure 1: Performance Overview across Heterogeneous GUI Benchmarks. We compare Step-GUI (4B/8B) against leading baselines on five diverse benchmarks covering grounding (ScreenSpot-Pro, OSWorld-G, MMBench-GUI-L2) and end-to-end agentic tasks (OSWorld, AndroidWorld). End-to-end results use pass@3 metric to mitigate non-model-related failures (e.g., CAPTCHA, VM crashes). Pass@1 results are also shown in a different shade of blue. The results demonstrate that Step-GUI-8B achieves state-of-the-art performance, outperforming existing open-source and proprietary agents, even surpassing models with much larger parameter counts. 1 Introduction The advent of Large Language Models (LLMs) unlocks unprecedented capabilities in natural language understanding and generation, fundamentally transforming how machines process and generate human language. Recent breakthroughs in multimodal large language models 6 ; 12 ; 32 ; 5 ; 18 further extend these capabilities to visual perception and action planning, providing the foundational abilities necessary for GUI automation. These advances enable models to understand complex visual interfaces and reason about appropriate interactions, opening new possibilities for autonomous GUI agents that can assist users in their daily digital tasks. Building on these foundational capabilities, the research community makes significant progress in GUI agent development. Various approaches explore methods for acquiring GUI interaction data 14 ; 39 ; 27 ; 46 , establishing critical data infrastructure for training GUI agents. Several powerful GUI agent models 35 ; 44 ; 59 ; 64 demonstrate impressive capabilities in navigating and manipulating graphical interfaces. However, a critical challenge remains: how to efficiently obtain high-quality trajectory and knowledge data to improve agent performance in target domains? Traditional annotation methods suffer from subjectivity and prohibitive costs, limiting the scalability of GUI agent development. To address this challenge, we introduce a self-evolving training pipeline centered on the Calibrated Step Reward System (CSRS). While prior work explores self-improvement through data generation via rationalization, self-critique, quality filtering, and offline reinforcement learning 63 ; 29 ; 19 ; 16 , these approaches primarily target single-turn tasks where model-generated annotations can be reliably verified. For multi-turn GUI agents where task execution spans multiple steps, direct model-generated annotations are prone to factual errors and hallucinations in the absence of verifiable ground truth. Unlike existing methods that rely solely on model introspection or post-hoc filtering, CSRS anchors LLM-generated dense step-level reasoning to trajectory-level evaluation signals, either automated verification scripts or human annotations, thereby ensuring annotation quality while maintaining scalability. Through trajectory-level calibration and LLM-powered knowledge extraction, CSRS converts model-generated trajectories into high-quality training data, achieving > 90% annotation accuracy with 10-100 \times cost reduction compared to traditional step-level annotation. Our progressive three-stage training paradigm orchestrates parallel data flows for novel exploration and strategic knowledge filtering, continuously improving model capabilities across multiple training rounds. On top of this pipeline, we train Step-GUI , a family of GUI specialist models (4B, 8B) built upon the Qwen3-VL backbone, and Step-GUI-8B achieves state-of-the-art performance: 80.2% on AndroidWorld, 48.5% on OSWorld, 65% on ScreenShot-Pro, and 89.91%/52.50% on our AndroidDaily benchmark (static/end-to-end). Remarkably, our compact 4B model delivers competitive performance while remaining deployable on consumer-grade hardware, enabling local execution without cloud dependencies. As GUI agents gain enhanced capabilities in visual understanding and autonomous task execution, two fundamental challenges emerge: standardized communication across heterogeneous devices and user privacy protection when processing sensitive data . The fragmentation of development efforts and high integration costs historically suppress innovation in tool ecosystems 36 . To address the standardization challenge, the Model Context Protocol (MCP) 2 is proposed as a universal standard that defines how agents and tools should communicate, enabling an interoperable “plug-and-play” ecosystem. Recent work explores how agents can effectively retrieve and select appropriate MCP services from large pools 13 and how to convert existing software projects into MCP format 34 . Building on these foundations, we propose GUI-MCP (Graphical User Interface - Model Context Protocol), the first MCP implementation specifically designed for GUI automation that addresses both standardization and privacy protection. GUI-MCP provides a standardized, cross-platform protocol that abstracts device capabilities into a small set of atomic and composite tools. Its hierarchical dual-layer architecture combines a Low-level MCP that offers fine-grained operations (e.g., clicks, swipes, text input) with a High-level MCP that delegates entire tasks to locally deployed GUI specialist models such as Step-GUI-4B. This design allows the main LLM to focus on high-level planning while it offloads routine GUI operations to local models. Critically, GUI-MCP supports a high-privacy execution mode in which raw screenshots and sensitive states stay on the device and only semantic summaries flow to external LLMs, which effectively protects user privacy while it leverages cloud-based reasoning capabilities. As the community makes progress on these technical challenges, attention is shifting toward a more pressing concern: how to evaluate whether GUI agents can handle real-world everyday usage beyond synthetic setups? Existing work develops diverse evaluation benchmarks, ranging from static benchmarks 10 ; 53 ; 22 ; 21 ; 8 ; 37 ; 66 ; 24 that assess element grounding and task planning through prediction-annotation matching, to interactive benchmarks 38 ; 55 that enable task execution in realistic operating system environments. However, these efforts either concentrate on single-step prediction accuracy or lack coverage of high-frequency Chinese mobile applications, leaving a critical gap: can agents reliably handle the high-frequency, everyday tasks that constitute real-world mobile usage? To address this gap, we introduce AndroidDaily , a benchmark explicitly grounded in empirical analysis of authentic mobile usage patterns. Rather than pursuing maximal application coverage, AndroidDaily focuses on ubiquitous daily scenarios (Transportation, Shopping, Social Media, Entertainment, Local Services) where agent deployment has immediate practical impact. The benchmark employs a two-tier evaluation strategy: a Static Benchmark with 3146 actions for efficient single-step action prediction, and an End-to-End Benchmark with 235 tasks that span multiple dimensions (scenario, task type, complexity, and ambiguity) and evaluate autonomous task completion in fully functional environments. Step-GUI-8B demonstrates strong performance on AndroidDaily, which highlights both its current capabilities and remaining challenges in realistic deployment scenarios. In