Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents

Ruiying Ma 1† , Shreya Shankar 1† , Ruiqi Chen 2 , Yiming Lin 1 , Sepanta Zeighami 1 , Rajoshi Ghosh 3 , Abhinav Gupta 3 , Anushrut Gupta 3 , Tanmai Gopal 3 , Aditya G. Parameswaran 1 · 2026

Read Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents by Ruiying Ma 1† , Shreya Shankar 1† , Ruiqi Chen 2 , Yiming Lin 1 , Sepanta Zeighami 1 , Rajoshi Ghosh 3 , Abhinav Gupta 3 , Anushrut Gupta 3 , Tanmai Gopal 3 , Aditya G. Parameswaran 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20576.txt.

Introduction

Ma and co-authors build a benchmark for agents that answer questions over company data. Real data sits split across many systems, with copies that clash. Key facts often hide in plain text, not tables. Their Data Agent Benchmark holds 104 queries over 17 datasets. Four database systems are in play. The best agent scored just 57 percent on first try. They study how agents fail and draw lessons for future work.

Catalog record

CREATORRuiying Ma 1† , Shreya Shankar 1† , Ruiqi Chen 2 , Yiming Lin 1 , Sepanta Zeighami 1 , Rajoshi Ghosh 3 , Abhinav Gupta 3 , Anushrut Gupta 3 , Tanmai Gopal 3 , Aditya G. Parameswaran 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99224
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20576

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents Thanks: † Co-first authors. Corresponding authors: { shreyashankar,adityagp } @berkeley.edu Ruiying Ma 1† , Shreya Shankar 1† , Ruiqi Chen 2 , Yiming Lin 1 , Sepanta Zeighami 1 , Rajoshi Ghosh 3 , Abhinav Gupta 3 , Anushrut Gupta 3 , Tanmai Gopal 3 , Aditya G. Parameswaran 1 Abstract . Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem—e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context—but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark ( DAB ), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB , the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench . Figure 1 . (a) In DAB , an agent solves a user task by interacting with database querying and Python execution tools within a ReAct-style loop. (b) In this example, the agent operates over unstructured text (i.e., extracting language from the details column in the 3rd tool call) and integrates data across different databases (PostgreSQL and SQLite) by reconciling the ill-formatted join keys (i.e., bref and bid , in the 5th tool call). (a) DAB setup. (b) Example agent trace. 1. Introduction Users across enterprises increasingly want data agents , or AI agents that answer natural-language questions over their data. Database vendors have begun adding agent capabilities to their platforms ( 41 ; 12 ) , and organizations are investing heavily in building their own: for example, Uber’s QueryGPT handles over 1.2 million interactive queries per month ( 44 ) , and OpenAI built an internal data agent used by thousands of employees to query 70,000 datasets totaling 600 petabytes ( 45 ) . Yet building reliable data agents remains difficult, because enterprise data is typically fragmented across multiple databases—surveys find that 72% of organizations store data in disparate silos ( 42 ) and 82% report that these silos disrupt critical workflows ( 17 ) —and answering a single question often requires integrating and reasoning across several of them. For example, consider a sales analyst who asks, “Which leads from last quarter should we follow up on?” Answering this requires finding lead records in a customer relationship management (CRM) tool, matching them against call transcripts stored in a separate document database, classifying each lead’s intent from unstructured text, and applying domain knowledge about what makes a lead good to pursue—all within a single agent session. Currently, no benchmark measures end-to-end data agent capabilities. For example, text-to-SQL benchmarks ( 27 ; 25 ; 8 ) test whether LLMs can translate a natural-language question into a single correct query over a single relational database, but do not require multi-step reasoning or integration across different databases. Or, table question-answering (i.e., Table-QA) benchmarks ( 9 ; 10 ) test reasoning over tables provided directly in the prompt, but production tables rarely fit in context and must be queried from databases directly. Overall, without an end-to-end benchmark, we cannot systematically identify where data agents fail or what capabilities most need improvement. A New Benchmark for Data Agents. To this end, we present Data Agent Benchmark ( DAB ), the first benchmark for evaluating AI agents on realistic, complex data-oriented tasks. To ensure DAB reflects production workloads, we conducted a formative study of query patterns from enterprise customers of PromptQL ( 36 ) —an organization building a production data agent—across six industries (technology, finance, food services, e-commerce, SaaS, and healthcare). We collected example queries that users posed to data agents, along with descriptions of their schemas, the database systems they used, and how their data was organized across them. From this study, we identified four properties that consistently make real-world data queries difficult and that are unaddressed by existing benchmarks: (i) multi-database integration : a single question may require querying across several databases with different query languages dialects (e.g., SQL dialects and MongoDB’s query language); (ii) ill-formatted join keys : identifiers for the same entity may differ across databases—e.g., through inconsistent prefixes, trailing whitespace, or abbreviated names—requiring the agent to detect and reconcile mismatches before joining; (iii) unstructured text transformation : answers may be embedded in text fields that the agent must parse into structured values before they can be filtered, grouped, or joined; and (iv) domain knowledge : answering the query correctly requires expertise not inferable from the data alone, such as knowing that stock volatility must be computed from adjusted closing prices to account for splits and dividends. Translating the aforementioned properties into a reproducible benchmark required careful design. Our benchmark, DAB , comprises 54 natural-language queries across 12 datasets, spanning 9 domains and 4 database management systems (DBMSes). Since enterprise data from the formative study is proprietary, we build DAB from open-source datasets across domains that match those observed in the formative study. These datasets are not inherently messy—the challenge was to systematically perturb them so that they exhibit the same characteristics we observed in production. For each dataset, we distribute data across at least two database systems (from PostgreSQL, MongoDB, SQLite, or DuckDB), mirroring how users in the formative study organize their data across heterogeneous backends. We then induce the remaining properties by often removing columns that would trivially answer a query and preserving their values in other forms that require more work to recover: reformatting join keys so that identifiers for the same entity differ across databases, and embedding structured attribute values into free-text fields that the agent must parse. Getting these perturbations right—realistic enough to be challenging, yet preserving deterministic ground-truth answers derived from the original data (not from LLM-generated or human judgments)—required substantial manual effort across all 12 datasets. Every query, answer, and dataset is verified by the authors. The size of DAB is comparable to other carefully curated and widely-adopted benchmarks (e.g., FrontierMath ( 13 ) , TerminalBench ( 30 ) ). Evaluating Frontier LLM Agents on DAB. Then, to characterize how agents perform on DAB , we evaluate a mix of proprietary and open-source frontier LLMs—GPT-5.2, GPT-5-mini, Gemini-3-Pro, Gemini-2.5-Flash, and Kimi-K2—using the ReAct pattern; a state-of-the-art agent architecture in which the model iteratively reasons about what to do next, issues a tool call (e.g., a database query or Python script), observes the result, and decides on the next action ( 48 ) . Each agent is equipped with tools for listing the databases available, executing queries against them, running Python code, and returning a final answer. An example agent trajectory is depicted in Figure 1 . For each query, we run 50 independent trials per agent and measure accuracy using pass@ k

Read the complete text