ProgRM : Build Better GUI Agents with Progress Rewards

Danyang Zhang ​, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen ​, Kai Yu · 2025

Read ProgRM : Build Better GUI Agents with Progress Rewards by Danyang Zhang ​, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen ​, Kai Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18121.txt.

Introduction

Danyang Zhang and seven co-authors improved in 2025 how screen-operating AI agents learn from their own practice runs. Such agents click through apps step by step, but training data is scarce and final success-or-failure grades waste the good steps inside failed attempts. Their progress reward model scores each step by how far it moves the task along.

A smart self-labeling method finds the key steps in past runs without human graders. Dense, fair feedback trains agents that recover from detours instead of being punished for them. Better training signals could make software-operating agents dependable enough for daily chores.

Catalog record

CREATORDanyang Zhang ​, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen ​, Kai Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProgRM : Build Better GUI Agents with Progress Rewards Danyang Zhang ​, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen ​, Kai Yu Abstract LLM-based (Large Language Model) GUI (Graphical User Interface) agents can potentially reshape our daily lives significantly. However, current LLM-based GUI agents suffer from the scarcity of high-quality training data owing to the difficulties of trajectory collection and reward annotation. Existing works have been exploring LLMs to collect trajectories for imitation learning or to offer reward signals for online RL training. However, the Outcome Reward Model (ORM) used in existing works cannot provide finegrained feedback and can over-penalize the valuable steps in finally failed trajectories. To this end, we propose Prog ress R eward M odel ( ProgRM ) to provide dense informative intermediate rewards by predicting a task completion progress for each step in online training. To handle the challenge of progress reward label annotation, we further design an efficient LCS-based (Longest Common Subsequence) self-annotation algorithm to discover the key steps in trajectories and assign progress labels accordingly. ProgRM is evaluated with extensive experiments and analyses. Actors trained with ProgRM outperform leading proprietary LLMs and ORM-trained actors, illustrating the effectiveness of ProgRM . The codes for experiments will be made publicly available upon acceptance. 1 Introduction Automatic Graphical User Interface (GUI) agents have great potential to reshape our daily lives by automating routine operations on GUI systems like computers and smartphones. Recently, surprised by the exceptional achievements of Large Language Models (LLM) in common-sense and reasoning domains, LLMs have been explored to improve the performance of GUI agents [ 37 , 36 , 24 , 35 , 40 , 20 , 43 , 2 , 16 , 30 , 17 , 31 , 13 , 8 , 11 , 28 ] . Using off-the-shelf LLMs to perform GUI tasks through prompt-based methods often yields unsatisfactory results, as these models lack the ability to ground instructions to low-level actions or to make long-term decisions required for GUI tasks. While there has been a growing body of work on training LLMs to build GUI agents, such training typically relies on human-labeled tasks or trajectories. However, annotating GUI tasks is labor-intensive, demands domain-specific expertise, and is extremely difficult to scale. The scarcity of high-quality training data presents a major challenge in developing high-performance GUI agents. To address this challenge, many studies have explored ways to automatically synthesize GUI training data [ 42 , 22 , 14 ] . These approaches primarily leverage LLMs to generate new task instructions and collect trajectories produced by actor LLMs. NNetnav [ 12 ] and OS-Genesis [ 23 ] conclude task instructions after collecting trajectories, while Explorer [ 14 ] further iterates on this process. However, these methods rely on imitation learning (see fig. 1 (a)) and are not well-suited for online environments, where content changes dynamically and agents may encounter unseen scenarios. They lack the ability to learn from mistakes or benefit from online exploration to improve performance. Efficient online Reinforcement Learning (RL) requires reliable reward signals to accurately evaluate arbitrary GUI-based tasks. DistRL [ 26 ] employs an additional Vision-Language Model (VLM) as an autonomous evaluator to determine task success, while WebRL [ 16 ] trains a lightweight Outcome Reward Model (ORM). However, these methods rely solely on the final success status, unnecessarily penalizing all steps within trajectories that fail to achieve the goal but potentially include valuable intermediate actions [ 8 ] , as shown in fig. 1 (b). Additionally, the sparse reward signals from ORM significantly reduce exploration efficiency in RL training, particularly for long-horizon tasks typical in GUI interactions. Figure 1: Comparison of policy optimization methods. (a) Imitation Learning optimizes the agent’s policy using per-step expert labels. (b) ORM provides sparse rewards by updating the policy only based on the trajectory’s final success or failure. (c) ProgRM predicts progress value at each step, using the progress gain ( \Delta_{p} ) as dense reward signal. To this end, we introduce the Prog ress R eward M odel ( ProgRM ), an effective and efficient approach for generating reward signals at intermediate steps within RL training trajectories for GUI agents. Intuitively, a complex task is not accomplished in a single leap; instead, it is completed through a sequence of subtasks that progressively advance toward the final goal. As shown in Figure 1 (c), at each stage, we can quantify how much of the overall task has been completed — this is the essence of progress . For example, when booking a flight, the process typically involves searching for available flights, selecting a suitable option, entering passenger details, and completing the payment. Each of these steps brings the agent closer to the goal, and ProgRM provides informative reward signals by estimating the progress achieved at each step, rather than waiting until the entire task is finished. This dense, intermediate reward signal enables more efficient and stable RL training, especially for long-horizon tasks commonly encountered in GUI environments. However, obtaining accurate progress reward labels poses a significant challenge for training reliable ProgRM models, as gold labels are commonly not directly available in raw trajectories. This problem is conceptually similar to the difficulties of process reward annotation faced in Process Reward Model (PRM) training in reasoning tasks. Human experts or Monte-Carlo search are commonly employed in the reasoning area to label rewards for intermediate reasoning steps [ 10 , 25 , 39 ] . However, such methods are prohibitively expensive and time-consuming for GUI agent tasks, where heavy simulators are involved and rapid rollback or efficient state restoration is often unsupported. Therefore, to address the progress reward labeling challenge, we propose an efficient self-annotation algorithm that automatically identifies key steps within trajectories and assigns progress labels accordingly. Specifically, for a given task, we first discover the common patterns from the successful trajectories by computing their Longest Common Subsequences (LCS) and extract them as execution recipes . The recipes are then used to identify the key steps in unseen trajectories, and the progress labels can be efficiently assigned based on the identified key steps. The proposed progress labeling algorithm prevents expensive human-expert annotation and Monte-Carlo search, while it can easily and sufficiently exploit the self-explored trajectories of agents. We evaluate the effectiveness of ProgRM on WikiHow task set [ 38 ] , a real-world Android device navigation benchmark. Experimental results demonstrate that ProgRM -trained actors outperform leading proprietary LLMs for GUI tasks, including Claude-3.7-Sonnet, achieving significant superiority in success rate. It also surpasses existing imitation learning and ORM-based online RL approaches. Furthermore, we show that ProgRM accurately captures the progresses agents make during navigation, highlighting its capability of providing meaningful intermediate feedback in complex environments. The key contributions of our work are as follows: • We propose ProgRM , a novel method that provides dense reward signals based on predicted progresses toward a goal, enabling effective online RL training for GUI agents. • We introduce an efficient LCS-based self-annotation algorithm to automatically generate progress labels for training ProgRM . • Experimental results on real-world GUI benchmark demonstrate the superiority of ProgRM -trained actors to state-of-the-art proprietary models as well as imitation-learning and ORM-training agents. 2 ProgRM : Progress reward model for

Read the complete text