An Efficient Training Pipeline for Reasoning Graphical User Interface Agents

Georgios PantazopoulosThe Alan Turing InstituteHeriot-Watt University [email protected] Eda B. ÖzyiğitThe Alan Turing Institute [email protected] · 2025

Read An Efficient Training Pipeline for Reasoning Graphical User Interface Agents by Georgios PantazopoulosThe Alan Turing InstituteHeriot-Watt University [email protected] Eda B. ÖzyiğitThe Alan Turing Institute [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08172.txt.

Introduction

Georgios Pantazopoulos and Eda Ozyigit describe a lean training pipeline for interface agents that must locate screen elements from language. Visual grounding lets an agent find the button or field a user request refers to. Many current methods depend on huge, noisy synthetic data sets, which are expensive to build and train on.

The pipeline filters nearly five million synthetic examples down to twelve thousand clean, diverse cases by removing misaligned items and keeping hard ones. A three billion parameter vision language model is then trained three ways, including reinforcement learning. The compact models match or beat larger baselines on several interface benchmarks.

Catalog record

CREATORGeorgios PantazopoulosThe Alan Turing InstituteHeriot-Watt University [email protected] Eda B. ÖzyiğitThe Alan Turing Institute [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Efficient Training Pipeline for Reasoning Graphical User Interface Agents Georgios PantazopoulosThe Alan Turing InstituteHeriot-Watt University [email protected] Eda B. ÖzyiğitThe Alan Turing Institute [email protected] Abstract Visual grounding is the task of localising image regions from natural language queries and is critical for reasoning capable Graphical User Interface agents. Many existing methods rely on massive, noisy synthetic datasets. This work introduces an efficient training pipeline that combines model-based data filtering with parameter-efficient fine-tuning. From 4.8M synthetic examples, 12K clean and diverse instances are curated by first identifying challenging cases, removing misaligned and then selecting a diverse set of multimodal instances. On this data, a 3B-parameter Vision-Language Model is trained under three regimes: supervised fine-tuning, chain-of-thought- augmented fine-tuning, and reinforcement learning via Group Relative Policy Optimization. Models trained with the filtered data and lightweight training strategies match or surpass larger baselines on benchmarks such as ScreenSpot, Multimodal-Mind2Web, and AndroidControl. These results demonstrate that principled data curation and robust adaptation can rival large-scale training, enabling compact yet capable multimodal reasoning agents. 1 1 1 The code is available at https://github.com/alan-turing-institute/gui-agent * * footnotetext: Work done during the internship at The Alan Turing Institute. 1 Introduction Visual grounding aims to link a natural language expression to its corresponding region in an image 19 ; 42 ; 29 ; 27 ; 28 . This capability is foundational for emerging Graphical User Interface (GUI) agents, as advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) now enable these agents to execute natural-language instructions and control desktop, mobile, and web applications 44 ; 34 ; 47 . The core requirements are: multimodal perception to fuse visual and textual signals; and action execution to perform precise interface operations with mouse, keyboard, or touchscreen, commonly accompanied by lighweight planning and state tracking 36 . Reliable control begins with accurate grounding of instructions to specific GUI elements (e.g. buttons, icons) ( 5 ; 18 ; 26 ) . The main approach for developing such agents equipped with exceptional grounding capabilities is to fine-tune an existing VLM on a large-scale synthetic corpora ( 9 ; 40 ; 22 ; 30 ) . A typical pipeline first applies an off-the-shelf GUI element detector to produce candidate regions, then uses a VLM to generate referring expressions or step-level instructions tied to each region, yielding paired (screenshot, region, text) examples for supervision. This paradigm naturally leads to agents that perceive the environment exclusively through visual observations that match or surpass the performance of agents relying solely on text-based representations ( 9 ; 49 ; 11 ; 6 ) . However, naively fine-tuning requires large volumes of diverse data often exhibiting weak generalisation in complex and high-resolution professional settings. With the recent success of reinforcement learning (RL) as a post-training stage, concurrent efforts ( 25 ; 24 ; 24 ; 43 ) pivot to adopting RL by designing reward functions that promote more precise grounding and reasoning capabilities. Group Relative Policy Optimization (GRPO) ( 10 ) , has recently emerged as one of the highly effective RL post-training regimes by replacing heavy value models with a simple rule-based reward system. Furthermore, recent works ( 41 ; 21 ) challenge the claim that reasoning necessitates excessive amounts of training data demonstrating that reasoning can emerge with only a few clean and representative examples. On the other hand, employing synthetic pipelines for creating GUI training instances often result in noisy examples stemming from misaligned instruction-region pairs, low quality or repetitive instructions, and ambiguous regions often as a product of the detection algorithm. Taken together, these observations call into question the necessity of high-volume synthetic instances for developing highly-capable GUI agents. In this work, we examine whether a highly-curated training seed can deliver comparable or superior performance, and explore the impact of different training regimes for building such agents. More specifically, we begin with a collection of high-volume synthetic training examples, and using a model-based filtering approach, we derive a clean, representative, and diverse set of examples for desktop, web, and mobile applications. Figure 1 provides an overview of the stages of our filtering approach. From the initial pool of 4.8M examples, we select 12k demonstrations by determining challenging instances for the base VLM, and enforcing multimodal diversity, which yields about a compression rate of 400%. We provide the chain-of-thought (CoT) traces for the intended solution from a GUI agent and explore the impact of supervised fine-tuning and GRPO with simple sparse-based rewards. Our findings illustrate that our best-performing model matches or even surpasses the performance of similar and larger GUI agents both in terms of grounding capabilities as well as success rate in agentic evaluations. The main contributions of this work are: (i) a detailed filtering method for curating misaligned synthetic data stemming from data generation pipelines using the same base-model as a critic for GUI training instances; (ii) a unified framework that applies model based filtering to web-based synthetic data and training state-of-the-art VLMs that is surprisingly effective for GUI agents; and (iii) a thorough evaluation of grounding and agent capabilities in GUI interfaces (desktop, mobile, and web). The results demonstrate that our agent achieves performance, at least comparable, and often substantially better, than the state-of-the-art agents. 2 Related Work GUI agents LLMs and VLMs have significantly accelerated progress in the development of GUI agents. Early approaches relied on structured representations such as HTML or accessibility (a11y) trees ( 49 ; 11 ; 6 ) for grounding with labelled bounding boxes ( 45 ; 12 ) . However, text-based representations of these environments are often noisy and/or incomplete ( 9 ) while at the same time, encoding the structure of the interface raises significant computational overhead as opposed to its corresponding visual observation. ( 48 ) As a result, more recent work has shifted toward pixel-level, visually grounded agents that directly interpret screen-level observations, leveraging advances in visual encoders and multimodal reasoning to achieve robust performance in complex GUI environments ( 5 ; 9 ; 40 ; 22 ; 13 ; 33 ; 37 ) . Building upon these developments, frameworks such as UI-TARS ( 30 ) and AgentS2 ( 1 ) adopt modular or generalist-specialist architectures to improve reasoning, planning, and cross-platform generalisation ( 3 ; 38 ) . Nevertheless, the predominant reliance on supervised fine-tuning (SFT) 7 across these systems presents enduring challenges, including substantial data requirements, limited adaptability to unseen interfaces, and suboptimal scalability in dynamic GUI contexts ( 24 ; 43 ) . These limitations have motivated increasing interest in learning paradigms that can leverage interaction-based or self-supervised signals to achieve more flexible and data-efficient GUI understanding. Reinforcement Learning Reinforcement learning (RL) has recently emerged as a promising alternative to SFT for training GUI agents, offering a mechanism to optimise model behaviour through interaction-based feedback rather than static supervision 46 . Rule-based reinforcement fine-tuning (RFT) paradigms have demonstrated impressive generalisation across reasoning, code generation, and multimodal perception tasks by employing verifiable reward signals to guide model adaptation ( 16 ; 35 ; 8 ) . Extending this paradigm to GUI environments, concurrent works such as UI-R1 ( 24 ) and GUI-R1 ( 25 ) have pioneered the integration of RFT for high-level interaction reasoning and low-level grounding, achieving improved action prediction

Read the complete text