DABstep: Data Agent Benchmark for Multi-step Reasoning

Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, Thomas Wolf · 2025

Read DABstep: Data Agent Benchmark for Multi-step Reasoning by Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, Thomas Wolf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23719.txt.

Introduction

Alex Egg and five colleagues introduce DABstep, a benchmark for AI data agents, in a paper published in two thousand twenty-five. Real data analysis mixes coding with reading documentation and reasoning across sources. Many benchmarks test only fragments of that workflow.

DABstep holds over 450 challenges drawn from a real financial analytics platform. Each task needs iterative, multi-step problem solving. Answers come in a factoid format that software can grade automatically. Leading language-model agents performed poorly in the authors' tests. The best reached only about fifteen percent accuracy on the hardest tasks. The paper details the benchmark's design and analyzes how agents fail. A public leaderboard accompanies the release.

Catalog record

CREATORAlex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, Thomas Wolf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DABstep: Data Agent Benchmark for Multi-step Reasoning Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, Thomas Wolf Abstract We introduce DABstep , a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks. DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Each task demands an iterative, multi-step problem-solving approach, testing capabilities in data manipulation, cross-referencing multiple sources, and precise result reporting. The benchmark provides a factoid-style answer format with automatic correctness checks for objective scoring at scale. We evaluate leading LLM-based agents, revealing a substantial performance gap: even the best agent achieves only 14.55% accuracy on the hardest tasks. We detail our benchmark’s design, dataset composition, task formulation, evaluation protocol, report baseline results and analyze failure modes. DABstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis. Data & Code: huggingface.co/spaces/adyen/DABstep Data & Dataset Card: huggingface.co/datasets/adyen/dabstep † † footnotetext: \clubsuit Equal contribution. 1 Introduction Recent advances in large language models (LLMs) have enabled the development of autonomous agentic workflows, particularly showing promise for automating complex, multi-step tasks within domains like data science and software engineering. However, the evaluation of such agents, especially for data analysis, faces significant hurdles. Many existing benchmarks rely on synthetic tasks, overly simplistic evaluations, or subjective assessment methods (such as LLM-as-a-judge, known for biases), limiting their ability to accurately reflect the challenges encountered in real-world analytical scenarios and gauge true agent capabilities. To address these limitations, we introduce the Data Agent Benchmark for Multi-step Reasoning (DABstep) which comprises over 450 authentic data analysis tasks derived directly from operational workloads at Adyen. Distinctly, these tasks combine structured (e.g., CSV, JSON) and unstructured data (e.g., text, domain-specific documentation or complicated manuals), requiring agents to demonstrate technical data manipulation skills (spanning SQL, statistical analysis, coding), a deep understanding of contextual instructions, and the ability to plan hierarchically. Figure 1: System overview of DABstep’s agent-task interaction. The figure illustrates the core components: task input (questions), agent, execution environment, and output (answers). Two representative questions are shown: a Risk/Fraud question from the Easy set (top), requiring 2+ data sources and at least 3 execution steps; and a Scheme Fees question from the Hard set (bottom), requiring 3+ sources and more complex reasoning over at least 6 steps. Agents must combine contextual understanding, code execution, and iterative refinement to produce a correct answer. See full trace example in section A.4 A core design principle of DABstep is its focus on multi-step reasoning complexity. Unlike benchmarks where tasks might be solvable via single-shot generation [ 47 , 24 , 21 , 20 ] , hard tasks in DABstep are designed to require multi-step reasoning . Agents must decompose problems into sequential, iterative steps—such as filtering data, computing aggregates, consulting reference tables, and handling intermediate results—often requiring interaction with a code execution environment as shown in Figure 1 . Furthermore, DABstep is designed for accessibility with a low barrier to entry, avoiding complex scaffolding or environments. An automated online leaderboard facilitates easy submission and standardized evaluation while also fostering community participation. Our evaluations underscore DABstep’s significant challenge to current state-of-the-art LLMs. Top-performing agents, such as o4-mini [ 34 ] , at time of writing, achieve only 14.55% accuracy (Table 1 ), highlighting substantial gaps between current agent capabilities and the demands of rigorous, practical data analysis. This paper makes the following key contributions: • Data : A novel benchmark featuring over 450 real-world data analysis tasks designed to test complex, multi-step reasoning and planning while leveraging diverse data sources including a large (+100k) payments dataset among others. • Factoid Evaluation Framework : An objective and standardized evaluation methodology centered on factoid answers with binary (right/wrong) outcomes, supported by a flexible scoring mechanism to handle formatting variations fairly . • Baselines : Performance results and failure modes for leading open and closed LLM agents, identifying current limitations and key areas for future research. • Community Platform : An accessible setup including a developer set, a quick-start notebook, an open-source baseline code, and a centralized live leaderboard to track progress and foster collaboration. Through DABstep, we aim to drive progress in developing AI agents capable of rigorous, practical, multi-step data analysis, better aligned with real-world analytical needs. 2 Design Principles & Related Works The guiding philosophy behind DABstep is grounded in four key principles that collectively emphasize realism, complexity, objectivity, and accessibility. In Table 2 we provide a high-level comparison to other related benchmarks on the main characteristics that we consider required to evaluate real-world data analysis tasks. Table 1: Performance of baseline models on the DABstep benchmark (Hidden Test Set). Scores reflect accuracy (%) on Hard and Easy splits. Costs are estimates based on public pricing at the time of evaluation (Q1 2025) and token usage; open models run locally are considered free (‘-’). All baselines run for a maximum of 10 steps per task with a ReAct style prompt except for the reasoning models. See Section 4 for methodology. Name Hard (%) Easy (%) Total Cost ($) o4-mini [ 34 ] 14.55 76.39 93 Claude 3.7 Sonnet [ 4 ] 13.76 75.00 139 o3-mini [ 34 ] 13.76 72.22 85 Gemini 2.5 Pro [ 10 ] 12.70 66.67 270 GPT 4.1 [ 33 ] 12.43 80.56 155 o1 [ 32 ] 11.11 69.44 435 Deepseek R1 [ 12 ] 11.04 68.21 3 Claude 3.5 Sonnet [ 3 ] 9.26 77.78 97 Llama 4 Maverick [ 2 ] 8.73 75.00 - GPT-4o [ 30 ] 6.08 66.67 53 Deepseek V3 [ 26 ] 5.56 66.67 2 Claude 3.5 Haiku [ 3 ] 5.03 77.78 35 Llama 3.3 70B [ 11 ] 3.70 68.06 - GPT-4o-mini [ 31 ] 3.44 69.44 3 Llama 4 Scout [ 2 ] 1.85 52.78 - Llama 3.2 1B [ 1 ] 0.00 1.39 - 2.1 Real-World and Multi-Step Complexity A central design principle of DABstep is its emphasis on realistic analytical challenges that require multi-step reasoning over heterogeneous data sources. Unlike benchmarks that focus on abstract math problems [ 14 ] , isolated code snippets [ 7 ] , or synthetic QA tasks [ 8 ] , DABstep’s over 450 tasks are derived directly from operational workloads at Adyen, reflecting the complex, iterative problem-solving scenarios faced by professional data analysts. These tasks are grounded in real-world financial analysis and integrate both structured data (e.g., CSV tables like payments.csv , JSON files like fees.json ) and unstructured documentation (e.g., Markdown files like manual.md ). Solving them demands technical proficiency in data manipulation (e.g., filtering, aggregation, joins), statistical reasoning, and the ability to extract and apply domain-specific rules from documentation. For example, agents must answer questions such as “Which card scheme had the highest average fraud rate in 2023?” or perform scenario analysis like “If merchant X changed its business category, how would that affect fees?” Crucially, DABstep tasks are explicitly designed to resist one-shot code generation strategies [ 47 , 24 ] . Especially in the Hard split (84% of tasks), no question can be answered through a single direct execution. Hard tasks require iterative data filtering and cross-referencing, which single-shot code

Read the complete text