HATS : Hardness-Aware Trajectory Synthesis for GUI Agents

Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen · 2026

Read HATS : Hardness-Aware Trajectory Synthesis for GUI Agents by Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12138.txt.

Introduction

Shao and co-authors synthesize training trajectories for GUI agents with an eye on hard actions. Some clicks carry meanings that depend on context, sequence, or a confusing screen. Current datasets underplay such actions, so agents trained on them misread real tasks. Their HATS framework scores hardness as the degree of semantic ambiguity in an action. One module steers data collection toward hard but useful interactions. A second module checks and repairs how instructions line up with execution, in a closed loop. Agents trained this way beat strong baselines.

Catalog record

CREATORRui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86920
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HATS : Hardness-Aware Trajectory Synthesis for GUI Agents Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen Abstract Graphical user interface (GUI) agents powered by large vision–language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantic-ambiguous actions—interactions whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS , a H ardness- A ware T rajectory S ynthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) a hardness-driven exploration that guides data collection toward ambiguous yet informative interactions, and (2) an alignment-guided refinement that iteratively validates and repairs instruction–execution alignment. The two modules operate in a closed-loop manner—exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments. † † footnotetext: † Equal contribution ∗ Corresponding author Figure 1 : Overview of trajectory synthesis paradigms. Compared with (a) existing methods, (b) HATS integrates hardness-driven exploration and alignment-guided refinement in a closed loop, producing high-quality trajectories with rich semantic coverage and strong instruction–execution alignment. (c) Experiments show HATS outperforms OS-Genesis by 100%↑ on AndroidWorld ( 22.60 vs. 11.30 ) and 215%↑ on WebArena ( 20.60 vs. 6.53 ). 1 Introduction Graphical User Interface (GUI) agents powered by large Vision–Language Models (VLMs) have emerged as a promising direction for automating complex digital workflows, from booking travel in mobile apps to configuring cloud services in browsers. A capable agent must perceive the screen, interpret a natural-language goal, decompose it into a plan, and execute actions such as Tap , Scroll , or Type . Training such autonomy requires instruction-aligned trajectories coupling task goals, step-level descriptions, and exact action–state pairs ( Zheng et al. 2024c ) . However, collecting such trajectories manually is prohibitively expensive, motivating recent efforts toward automated trajectory synthesis. Automating Trajectory Synthesis. To reduce human annotation, recent work automates trajectory generation. Early task-driven approaches begin with written goals and record agents or humans executing them ( Li et al. 2024 ; Lù et al. 2024 ; Lai et al. 2024 ) , but still depend on costly goal specification and yield inconsistent traces ( Murty et al. 2024 ; Patel et al. 2024 ) . OS-Genesis ( Sun et al. 2025 ) instead introduces reverse task synthesis , where the system explores applications via random UI actions and then synthesizes task instructions to describe observed outcomes, improving diversity by removing manual goal authoring. The Semantic-Ambiguity Gap. Despite these advantages, reverse task synthesis ( Sun et al. 2025 ) still faces critical limitations. Exploration based on uniform random walks or shallow breadth-first search causes over 70% of collected traces to collapse into trivial semantic-intuitive actions (e.g., “open menu”, “tap back”), which quickly saturate learning yet dominate storage and training budgets. Conversely, the resulting corpora remain deficient in semantic-ambiguous actions —UI interactions whose functional meaning depends on subtle visual or contextual cues and are easily confused or hidden. As illustrated in Fig. 2 , these actions include (a) identical “plus” icons that trigger different functions depending on context, (b) operations that only succeed after prerequisite steps, and (c) visually similar elements leading to distinct outcomes. Such rare yet crucial actions represent the key bottlenecks where agents fail to generalize: without adequate exposure, they plateau at superficial behaviors and cannot handle unseen workflows. Moreover, when these actions do appear, their ambiguity often leads to instruction–execution misalignment : synthesized instructions may omit essential context or replay inconsistently, introducing noisy supervision that further degrades data quality. This issue is exacerbated by existing one-shot instruction generation methods ( Sun et al. 2025 ) , which lack semantic validation and thus produce corpora that are both redundant and unreliable. The challenge is thus twofold: (i) how to focus exploration on high-value, semantically ambiguous regions of the state space rather than collecting redundant, intuitive actions; and (ii) how to ensure alignment between synthesized instructions and actual executions to prevent supervision noise. Existing one-shot instruction generation lacks semantic validation, making the resulting corpora both redundant and unreliable. Figure 2 : Illustrative cases of semantic-ambiguous actions. Such actions constitute critical bottlenecks for robust agent generalization but are rarely captured by existing synthesis pipelines. Hardness-Aware Trajectory Synthesis (HATS). To address these challenges, we propose HATS , a closed-loop trajectory synthesis framework explicitly targeting semantic-ambiguous actions through two cooperative modules (Fig. 1 ). The Hardness-Driven Exploration module (§ 3.1 ) steers search toward semantically complex, under-represented states, while the Alignment-Guided Refinement module (§ 3.2 ) performs multi-round replay and repair to ensure semantic consistency before data admission. Both modules operate under a unified Hardness-Driven Monte Carlo Tree Search (HD-MCTS) procedure, which orchestrates exploration, refinement, and feedback propagation in a single control loop. Exploration supplies challenging trajectories for validation, and refinement converts misalignment feedback into hardness signals that guide subsequent exploration. This closed-loop design systematically enhances both the diversity and fidelity of synthesized data, yielding more reliable and generalizable GUI agents. Figure 3 : Architecture of the HATS framework. The framework integrates a Hardness-Driven Exploration module (§ 3.1 ) and an Alignment-Guided Refinement module (§ 3.2 ) within a unified HD-MCTS loop. Exploration corresponds to the Selection , Expansion , and Simulation Phase I , while refinement handles Simulation Phase II and Backpropagation . Misalignment detected during refinement is converted into a hardness reward that guides subsequent exploration, forming a closed loop for progressively improving both diversity and semantic fidelity of synthesized trajectories. Contributions. We summarize our contributions as follows: • I. We introduce HATS , a unified closed-loop trajectory synthesis framework that enhances the representation and handling of semantic-ambiguous actions. • II. We design a hardness-driven exploration module that prioritizes rare, high-value actions while reducing redundancy from intuitive ones. • III. We propose an alignment-guided refinement module that replaces one-shot synthesis with multi-round validation, ensuring instruction–execution alignment. These contributions collectively establish a scalable, semantically grounded pipeline that enhances diversity and fidelity in trajectory synthesis, advancing GUI agent training. 2 Related Works GUI Agents. Recent advances in large language and vision–language models (LLMs/VLMs) have accelerated progress in agents that perceive, reason, and act within graphical user interfaces ( Durante et al. 2024 ; Feng et al. 2024 ; Wu et al. 2024 ) . Earlier systems relied on prompt programming or external tool execution ( Wu et al. 2023 ; Sun et al. 2024 ) , while more recent work fine-tunes multimodal backbones to better recognize UI states ( Cheng et al. 2024 ; Wu et al. 2025 ) and execute atomic actions across web, desktop, and mobile environments ( Kapoor et al. 2024 ; Li et al. 2024 ; Wang et al. 2024a ) .

Read the complete text