WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces
Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong · 2026
Read WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces by Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05295.txt.
Introduction
Sicheng Fan and colleagues introduce WebChain, the largest open dataset of human-annotated real-world website trajectories, accelerating web-agent research. It holds 31,725 trajectories and 318,000 steps, with visual, structural, and action Triple Alignment. Collection uses a scalable pipeline covering complex, high-value tasks that synthetic methods often miss. The authors propose a Dual Mid-Training recipe decoupling spatial grounding from planning, reaching state-of-the-art WebChainBench and public GUI performance. The work provides data and insights for building and evaluating next-generation web agents.
Catalog record
| CREATOR | Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32444 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong Abstract We introduce WebChain , the largest open-source dataset of human-annotated trajectories on real-world websites, designed to accelerate reproducible research in web agents. It contains 31,725 trajectories and 318k steps, featuring a core Triple Alignment of visual, structural, and action data to provide rich, multi-modal supervision. The data is collected via a scalable pipeline that ensures coverage of complex, high-value tasks often missed by synthetic methods. Leveraging this dataset, we propose a Dual Mid-Training recipe that decouples spatial grounding from planning, achieving state-of-the-art performance on our proposed WebChainBench and other public GUI benchmarks. Our work provides the data and insights necessary to build and rigorously evaluate the next generation of scalable web agents. 1 Introduction Conquering the browser is one of the most valuable problems in the field of GUI agents, as the web browser serves as the primary interface for the vast majority of digital tasks for human users. Recent advances in Vision-Language-Action (VLA) modeling have sparked hope for agents capable of precise spatial grounding, robust reasoning, and safe long-horizon decision-making 9 ; 20 . Due to the complex and dynamic nature of web environments and the inherent lack of trajectory data, there is a persistent need for up-to-date, large-scale, high-quality datasets in the web agent domain. Current data requirements range from multi-modality spatial grounding, which involves accurately identifying and interacting with UI elements based on visual and textual information, to long-horizon planning trajectories that enable agents to execute complex and multi-step tasks, are becoming increasingly refined with the diversification of evaluation methods and benchmarks. Existing data efforts face significant limitations. Open-source, human-annotated datasets often lack the necessary scale and completeness 6 ; 16 ; 25 (see Table 1 ), which obstructs the verification of model scaling effects. On the other hand, while data synthetic methods 19 ; 21 attempt to harvest real-world interactions at a low cost, they are inherently constrained by security mechanisms. These methods often crumble when facing anti-bot detection, CAPTCHAs, or scenarios requiring personal authentication (e.g., logging into a bank account), rendering them unable to capture the most valuable, high-value user workflows. Furthermore, a significant body of work on scaling GUI agent models relies on proprietary datasets 20 ; 19 ; 22 , derived from either manual annotation or synthetic methods. This practice renders their key insights non-reproducible, which severely impedes the establishment of community consensus and the overall advancement of the field. Table 1 : Comparison with Existing Web Interaction Trajectories Datasets Feature WebChain Mind2Web WebArena (Env.) WebLINX GUIAct(multi) Scale Trajectories 31725 2350 N/A 2337 5696 Steps 318k 17155 N/A 100k+ 44k Websites 428 137 4 Domains 155 121 Data Source Real Websites ✓ ✓ × ✓ ✓ Human Trajectories ✓ ✓ × ✓ ✓ Visual Data Viewport Screenshot ✓ ✓ ✓ ✓ ✓ Localization Pixel-level Coordinates ✓ ✓ ✓ ✓ ✓ Bounding Boxes ✓ ✓ × ✓ ✓ Accessibility Tree ✓ × ✓ × × Temporal Granularity Event Timestamps ✓ ✓ × ✓ × To democratize research in this field and break the data monopoly, we introduce WebChain , a fully open-source ecosystem comprising the largest scale web interaction trajectories collected from actual websites with human annotations. WebChain is totally constructed by human annotators operating on live, diverse websites, ensuring the inclusion of complex tasks that require authentication and nuanced decision-making. Central to our contribution is the Triple Alignment mechanism. Our logging pipeline rigorously synchronizes three layers of context: (1) Visual Context via viewport and full-page screenshots; (2) Structural Context via Accessibility (AX) trees; and (3) Action Alignment via precise pixel coordinates, bounding boxes, and CSS selectors. This multi-layered supervision allows models to not only ”see” the page but to understand the structural logic behind every pixel, providing rich signals for grounding, attention allocation, and DOM-aware navigation. Beyond the data itself, we demonstrate that WebChain serves as a fundamental building block for a new generation of web agents. We introduce a rigorous evaluation suite, WebChainBench , to measure progress in both spatial grounding and long-horizon planning. Through extensive experiments, we reveal critical scaling laws, showing that model performance on long-horizon tasks correlates positively with the scale of our human-verified data. Furthermore, we identify a superior training paradigm— Dual Mid-Training —which decouples spatial perception from temporal planning, achieving state-of-the-art (SOTA) performance on both WebChainBench and well-adopted public benchmarks. In summary, our contributions are three-fold: • Largest Human-Annotated Real-World Web Trajectory Dataset: We release the largest human-annotated dataset collected from real-world websites to date. We make the entire pipeline—including the data, collection tools, and benchmarks—publicly available to foster community innovation. • Scalable Construction Pipeline: We propose a robust pipeline combining constraint-based task synthesis, human-in-the-loop verification, and automated context enrichment (including CoT and visual grounding densification), enabling the efficient production of high-quality data at scale. • Investigation on Training Recipes & SOTA Performance: We validate the dataset’s value through extensive experiments to identify the optimal training recipe for scaling GUI agent models. Our findings reveal that when building upon a reinforcement finetuning foundation, a Dual Mid-Training paradigm—which disentangles spatial grounding from planning—significantly boosts performance. This approach achieves new state-of-the-art results on complex, long-horizon web tasks, outperforming standard supervised methods. Figure 1 : Dataset Overview. A statistical summary of WebChain, including interaction distribution across website categories, top-domain interaction frequencies, device pixel ratio distribution, trajectory complexity, and trajectory duration. These statistics collectively highlight the scale and diversity of WebChain. 2 Related Work 2.1 Web Interaction Datasets and Benchmarks The evolution of web agents is fundamentally constrained by the availability of training and evaluation environments. Early research relied on synthetic, sandboxed environments like MiniWoB++ 14 , which offered controlled tasks but failed to reflect the visual and structural noise of the open web. To bridge this gap, data-driven approaches emerged, mining interactions from real-world applications. Rico 5 pioneered this in mobile UIs, while Mind2Web 6 and WebLINX 16 extended it to the web, providing large-scale, offline trajectories. However, these datasets still lack the scale required to systematically validate the scaling effects of modern GUI agent models. To enable interactive evaluation, recent works have introduced realistic execution environments. WebArena 25 and VisualWebArena 12 host reproducible websites but suffer from the ”simulation gap”—lacking the evolving DOM structures, ads, and visual clutter of the live web. Conversely, synthetic methods like Explorer 19 and OS-Genesis 21 operate on the live web but are inherently limited by security boundaries; they cannot capture high-value workflows requiring authentication (e.g., banking, e-commerce checkout) due to bot detection and privacy constraints. WebChain fills this critical void by providing the largest, human-annotated corpus collected on live, diverse websites. Unlike synthetic methods, our human-in-the-loop pipeline captures complex, authenticated trajectories, while our Triple Alignment ensures distinct supervision for visual, structural, and action grounding. 2.2 Vision-Language Models for GUI Grounding The paradigm for GUI agents has shifted from DOM-based language models to pixel-aware Vision-Language-Action (VLA) models. Early approaches relied on parsing HTML text 8 , often struggling with the token context limits of complex web pages. Recent advancements, driven by general-purpose VLMs (e.g., GPT-4V 1 , Qwen2.5-VL 2 ) and specialized UI models (e.g., SeeAct 24 , CogAgent