WebDS: An End-to-End Benchmark for Web-based Data Science

Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty Tenghao Huang, Christopher D. Manning · 2025

Read WebDS: An End-to-End Benchmark for Web-based Data Science by Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty Tenghao Huang, Christopher D. Manning (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01222.txt.

Introduction

Ethan Hsu and co-authors build a benchmark that tests AI agents on data science the way it actually happens: on the open web. Real projects mean hunting for data online, merging material from many places and formats, and producing an analysis. Older web benchmarks use simple interactions, while classic data science tests hand over clean, static tables, so neither measures the full workflow.

Their benchmark, WebDS, holds 870 tasks spread over 29 websites, from government data portals to news outlets. Leading agents struggle badly: one that finishes eighty percent of a popular web benchmark completes only fifteen percent here. The failures are new in kind, including weak grounding in sources, looping behavior, and shortcut-taking. Humans score around ninety percent, which marks how far agents still have to go.

Catalog record

CREATOREthan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty Tenghao Huang, Christopher D. Manning
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WebDS: An End-to-End Benchmark for Web-based Data Science Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty Tenghao Huang, Christopher D. Manning Abstract A large portion of real-world data science tasks are complex and require multi-hop web-based interactions: finding appropriate data available on the internet, synthesizing real-time data of various modalities from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions, such as form submissions or e-commerce transactions, and often do not require diverse tool-using capabilities required for web based data science. Conversely, traditional data science benchmarks typically concentrate on static, often textually bound datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step operations requiring the use of tools and heterogeneous data formats that better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes 80 % 80\% of tasks on Web Voyager, successfully completes only 15 % 15\% of tasks in WebDS, which our analysis suggests is due to new failure modes like poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS’ tasks display. By providing a more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLM-based data science. † † footnotetext: * First Author † † footnotetext: † \dagger Second Author 1 Introduction The web offers a vast repository of knowledge, making automated web traversal a powerful tool for enhancing information retrieval, analytical workflows, and decision-making. Large language models (LLMs) have unlocked new frontiers in knowledge collection, enabling them to interact with websites, synthesize insights, and execute complex tasks with minimal human intervention hong2024data 1 1 1 For e.g, see recent advances like Deep Research tools by OpenAI (https://openai.com/index/introducing-deep-research/) and Google Deepmind . These agents hold significant promise for data science applications, facilitating autonomous data collection, synthesis, visualization, and predictive analytics peasley2025leveraging . By providing an intuitive natural language interface, LLM-powered agents can democratize access to advanced data analysis, making AI-driven insights more accessible across expertise levels jansen2025leveraging . Yet, extracting deep insights from websites is not trivial. Modern webpages are complex and dynamic, often containing scattered, unstructured, and multimodal data spread across multiple layers of interactivity. Navigating these environments autonomously presents significant challenges, as web agents must contend with varying page structures, access restrictions, dynamically generated content, and inconsistent data formats. Additionally, effectively synthesizing and analyzing retrieved information requires robust reasoning capabilities and contextual understanding. While web agents have shown promise in certain settings, particularly in following natural language instructions to accomplish simple tasks such as writing Reddit posts or buying items zheng2024gpt , prior evaluations narrowly targeted web browsing only , without assessing agents on their ability to manipulate and understand data webarena ; pan2024webcanvas . Existing web agent benchmarks face trade-offs limiting their long-term usefulness and precision. WebVoyager he2024webvoyagerbuildingendtoendweb , for instance, uses live sites that change over time, hindering reproducibility. Others li2020mapping ; deng2024mind2web ; webarena rely on coarse metrics that miss finer agent behaviors. Meanwhile, existing data science benchmarks primarily emphasize data manipulation within code-based environments or structured spreadsheets 2502.16395v1 , or focus on querying unstructured data from static databases rajpurkar2016squad . Yet in realistic workflows, data scientists and other knowledge workers often begin by browsing the web, navigating across multiple websites, and synthesizing information from heterogeneous sources. Thus, browsing, searching and synthesizing are a fundamental – yet often overlooked –components of real-world web-based data science tasks EnterpriseDataAnalysis ; sun2022supporting . To address the limitations of existing web agent and data science benchmarks, we introduce WebDS, a novel benchmark designed to reflect the full data science workflow—from data acquisition via web browsing to downstream analysis, visualization, reasoning, and reporting. We find that these tasks pose a significantly more challenging testing ground for leading models: for example, while GPT-4o with Browser Use obtains an 81.1\% success rate on WebVoyager browser_use2024 it only obtains 13.2\% on WebDS. Furthermore, surprisingly, we find that increasing model-capacity does not necessarily lead to increased performance: GPT-4o performs similarly to GPT-4o-mini and Qwen2.5-72b. An error analysis (Section 7 ) shows significant, novel failure modes for all models: poor information grounding (due to the numerical nature of data science tasks), repetitive behavior and shortcut-taking (due to the multi-hop nature of web-based data science). In sum, WebDS introduces a significantly more challenging, real-world benchmark that represents a next frontier for model development. By bridging the gap between web interaction and data science capabilities, we hope that our work grounds the community and pushes us towards developing novel end-to-end AI skills. Our contributions are threefold: 1. Comprehensive Task Suite We design a benchmark comprising of 870 tasks spanning a wider range of data types, modalities, and domains than any prior benchmark across 29 websites and 10 domains which we discuss more in section 4 and 3 . These tasks require not only analytical reasoning but also interaction with diverse tools and and interfaces. For example, "Is there an optimal level of healthcare spending to maximize life expectancy, excluding outliers? Write a policy brief to a think tank and a separate document detailing the mathematical analysis.” Solving this task entails identifying relevant data (e.g., from sites like Our World in Data), selecting and applying appropriate analytical techniques (such as nonlinear optimization), and translating technical findings into accessible, domain-specific insights. 2. Realistic End-to-End Evaluation WebDS is the first benchmark to assess the full data science pipeline, simulating end-to-end workflows. Tasks begin with autonomous web browsing for relevant data, followed by analysis and or visualization, and culminate in generating well-reasoned, context-aware outputs. 3. Reproducible and Fine Grained Evaluation Framework. To ensure experimental validity and comparability, we provide a containerized benchmark environment with controlled versions of websites, enabling robust replication. In contrast to prior work that relied on live web content or overly static settings, WebDS strikes a balance between reproducibility and task realism. We also introduce fine-grained metrics that capture subtask completion, tool usage accuracy, data validity, reasoning quality, and report fidelity—allowing for nuanced assessment of agent capabilities. Finally, we also include a diverse mix of end-to-end task and modular skill-specific test, enabling both holistic evaluation and pointed evaluation for targeted diagnostics. 2 Related Work Data Analysis Benchmarks Benchmarks such as SQuAD rajpurkar2016squad and HotpotQA yang2018hotpotqa have played critical roles in evaluating language models’ reasoning over structured and semi-structured data, providing passage-based and multi-hop question-answering tasks, respectively. Further, Wikitable pasupat2015compositional offers a challenge in executing compositional logical queries over semi-structured tables, testing models’ ability to generalize across complex tabular reasoning tasks. Recently, InfiAgent-DABench hu2024infiagent emerged as the first specialized benchmark for data analysis agents, comprising 311 tasks across 55 CSV files with automated evaluation. However, these benchmarks’ focus on structured datasets limits assessment of real-world iterative workflows: data science workflows often heavily involve finding data (e.g. through web navigation), using tools, and

Read the complete text