GTA : Generating Long-Horizon Tasks for Web Agents at Scale

Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu · 2026

Read GTA : Generating Long-Horizon Tasks for Web Agents at Scale by Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29218.txt.

Introduction

Tenghao Huang and colleagues present GTA, making long-horizon tasks for web agents with process supervision. Existing benchmarks are handmade, offering coarse start-goal notes without paths, while automatic efforts stay costly and shallow. GTA integrates crawling, search seeding, contextual generation, and quality control to produce tasks paired with ready trajectories. Decoupling crawling from generation improves efficiency, grounding tasks in the site graph enforces compositionality, and replays guide. On over 50 websites spanning shops, government, forums, and news, with multihop reach, the benchmark reveals a human-agent gap.

Catalog record

CREATORTenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GTA : Generating Long-Horizon Tasks for Web Agents at Scale Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu Abstract Web agents, which couple language models with browsing and tool-use capabilities, show promise as open web assistants. Yet progress is increasingly limited by the lack of scalable, process-level supervision. Existing benchmarks are largely manually constructed , providing only coarse start–goal annotations without intermediate trajectories, while recent automatic generation efforts remain expensive, biased, and shallow. These limitations prevent reliable training and evaluation of agents that must generalize to realistic, multi-hop, cross-page tasks. We introduce a scalable framework GTA that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control to produce realistic tasks paired with executable trajectories. This design decouples crawling from generation for greater efficiency, grounds tasks in the site graph to enforce compositionality, and ensures dense supervision through deterministic replays and systematic validation. We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage. The resulting benchmark reveals a significant human–agent performance gap and enables detailed diagnostics. Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a self-evolving benchmark ecosystem where end users can generate up-to-date tasks grounded in live web content 1 1 1 Code available at https://github.com/tenghaohuang/GTA . . 1 Introduction Web agents—language models coupled with a browser and tool-use interfaces—have rapidly emerged as a core direction for building general-purpose assistants that can search, navigate, and act on the open web ( 9 ; 7 ; 17 ; 3 ; 5 ) . Despite striking demonstrations, progress is increasingly constrained by data. Existing benchmarks provide only human-annotated task descriptions paired with a curated end-to-end solution, offering no supervision over the agent’s latent decision process ( 1 ; 12 ; 17 ; 11 ; 4 ) . This is problematic: prior work has highlighted that web navigation is naturally a hidden Markov process, where the critical uncertainty lies in the unobserved intermediate states and actions ( 10 ; 3 ) . Without reliable ground-truth trajectories, agents are overfitting to small hand-authored sets, and struggling to generalize to realistic multi-hop, cross-page use cases ( 8 ; 6 ) . To address the scarcity of large-scale data, recent efforts have begun to generate web tasks automatically. For instance, AgentTrek ( 15 ) converts online tutorial documents into executable trajectories and validates them with a vision-language agent, while WebWalker ( 14 ) and NNetNav ( 8 ) use an LLM-based policy model to explore websites and then label the resulting trajectories with intent. While promising, these approaches face three critical limitations. First, they are prohibitively expensive in both time and cost : constructing a single task requires repeated agent–environment rollouts and multiple LLM calls, driving costs upward and limiting throughput. Second, they induce strong exploration bias : LLM policies tend to overfit to “obvious” paths (e.g., repeatedly selecting product detail pages), while overlooking more nuanced or less salient site functionalities such as rewards programs, sales, or new arrivals. As a result, large portions of the site graph remain unexplored and untested. Third, they lack compositionality and utility : most tasks generated by existing approaches collapse to single-hop retrieval or form-filling that could be solved by a simple search query, failing to capture the multi-hop, cross-page dependencies of real-world navigation. As a result, such tasks offer little practical utility for users, since solving them does not require deeper reasoning, sustained interaction, or problem-solving skills that would actually test or benefit an intelligent web agent in realistic settings. We provide detailed comparison in Table 1 . We introduce GTA ( G enerate Long-Horizon T asks for Web A gents at Scale), a scalable task-generation and benchmarking framework for web agents that integrates crawl , retrieve , in-context generation , and quality control to produce realistic, graded-difficulty tasks paired with executable ground-truth trajectories . Concretely, we first crawl publicly accessible sites to construct a content/link graph; next, we retrieve candidate pages and evidence snippets as seeds; we then generate tasks via in-context prompting that references on-site facts, UI elements, and workflows; we filter with an LLM-based quality control protocol that enforces clarity and solvability against the current crawl; and finally we record a minimal, programmatically executable navigation path (clicks, scrolls, inputs) enabling deterministic replays and step-level attribution. This design directly targets the three challenges identified earlier. Efficiency and cost : we separate one-time crawling from lightweight, retrieval-seeded task generation, eliminating repeated policy rollouts and reducing both redundant LLM calls and wall-clock latency. Exploration coverage : by leveraging retrieval-based seeding rather than policy-driven exploration, our method avoids the strong biases of LLM navigators and systematically samples diverse, semantically rich regions of each website’s graph. Compositionality and utility : grounding generation in the site graph enables the construction of multi-hop tasks that require reasoning across interconnected pages, mirroring the dependencies and open-ended objectives characteristic of real user workflows. Dataset Multi-hop Scalable Auto-gen Groundtruth Dynamic Multilingual WebVoyager 12 ✗ ✗ ✓ ✗ ✗ WebArena 17 ✗ ✗ ✓ ✗ ✗ WebWalker 14 ✓ ✗ ✓ ✗ ✓ BrowseComp 13 ✓ ✗ ✓ ✗ ✗ WebDS 2 ✓ ✗ ✓ ✗ ✗ AgentTrek 15 ✗ ✓ ✗ ✗ ✗ NNetNav 8 ✓ ✓ ✗ ✗ ✗ GTA (Ours) ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of benchmark properties. GTA is the only benchmark that combines automatic task generation , multi-hop reasoning , executable ground-truth , dynamic expansion , and multilingual coverage . To automatically generate diverse multi-hop web agent tasks, we distinguish between two primary settings: intra-website queries , where reasoning unfolds entirely within a single domain, and inter-website queries , which require coordination across multiple domains. Within intra-website tasks, our benchmark spans a broad range of languages, including English, Italian, Japanese, German, and Chinese, ensuring coverage beyond English-centric environments. We instantiate the pipeline on over 50 public websites spanning e-commerce catalogs, entertainment/government pages, forums, and finances, etc. Beyond automatic filtering, we collect human annotations to assess answer correctness, complexity, utility and realism, confirming that tasks are reasonable for humans yet challenging for current agents. Our contributions are three-fold: • Problem formulation. We introduce and formally define the new task of multi-hop web-agent task generation , where tasks require agents to reason across multiple pages or domains rather than collapsing to trivial information lookup. • Efficient automatic data generation. We propose an efficient pipeline that markedly reduces per-task time and cost while producing challenging tasks; each task includes an executable minimal “gold path” for deterministic replay and step-level diagnostics, safeguarded via verifier checks and negative controls. • Evolving benchmark. Rather than treating GTA as a fixed dataset, we frame it as a self-evolving benchmark ecosystem in which end users can continually generate tasks from current web content. Coupled with our verification pipeline, GTA preserves task quality and reproducibility while exposing agents to a changing, realistic task distribution, thereby reducing overfitting and enabling ongoing evaluation of generalization. 2 Benchmark Design The goal of our benchmark is to create web-agent tasks that systematically capture the multi-hop nature of realistic online interactions. Unlike prior datasets that treat each task as a

Read the complete text