ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines
Tengjun Jin, Yuxuan Zhu, Daniel Kang · 2025
Read ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines by Tengjun Jin, Yuxuan Zhu, Daniel Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04808.txt.
Introduction
Tengjun Jin, Yuxuan Zhu, and Daniel Kang built a benchmark for AI agents that construct data pipelines in a 2025 study. Cloud warehouses made extract-load-transform pipelines common, yet designing them takes heavy manual care to keep data correct. Existing tests grade isolated skills, such as writing one transformation, not the whole pipeline an engineer must deliver.
Their end-to-end benchmark asks agents to build complete pipelines and checks the results. Pipeline work mixes tool use, schema reasoning, and debugging in sequence. A realistic grade tells companies whether agents can take over routine data engineering or only assist at the edges.
Catalog record
| CREATOR | Tengjun Jin, Yuxuan Zhu, Daniel Kang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines Tengjun Jin, Yuxuan Zhu, Daniel Kang Abstract . Practitioners are increasingly turning to Extract-Load-Transform (ELT) pipelines with the widespread adoption of cloud data warehouses. However, designing these pipelines often involves significant manual work to ensure correctness. Recent advances in AI-based methods, which have shown strong capabilities in data tasks, such as text-to-SQL, present an opportunity to alleviate manual efforts in developing ELT pipelines. Unfortunately, current benchmarks in data engineering only evaluate isolated tasks, such as using data tools and writing data transformation queries, leaving a significant gap in evaluating AI agents for generating end-to-end ELT pipelines. To fill this gap, we introduce ELT-Bench, an end-to-end benchmark designed to assess the capabilities of AI agents to build ELT pipelines. ELT-Bench consists of 100 pipelines, including 835 source tables and 203 data models across various domains. By simulating realistic scenarios involving the integration of diverse data sources and the use of popular data tools, ELT-Bench evaluates AI agents’ abilities in handling complex data engineering workflows. AI agents must interact with databases and data tools, write code and SQL queries, and orchestrate every pipeline stage. We evaluate two representative code agent frameworks, Spider-Agent and SWE-Agent, using six popular Large Language Models (LLMs) on ELT-Bench. The highest-performing agent, Spider-Agent Claude-3.7-Sonnet with extended thinking, correctly generates only 3.9% of data models, with an average cost of $4.30 and 89.3 steps per pipeline. Our experimental results demonstrate the challenges of ELT-Bench and highlight the need for a more advanced AI agent to reduce manual effort in ELT workflows. Our code and data are available at https://github.com/uiuc-kang-lab/ELT-Bench . 1. Introduction Data engineers are increasingly leveraging Extract-Load-Transform (ELT) pipelines to integrate data and efficiently transform it into the required format as scalable cloud data warehouses become more accessible and storage prices continue to fall ( Seenivasan, 2022 ; Mbata et al., 2024 ; Foidl et al., 2024 ; Singhal and Aggarwal, 2022 ; Simitsis et al., 2023 ) . For example, the TPC-DI benchmark requires the creation of a decision support system for a retail brokerage firm by transforming data from various sources, including a trading system, internal Human Resources (HR), and Customer Relationship Management (CRM) systems ( Poess et al., 2014 ) . These data sources vary in formats, data types, attributes, and inter-table relationships ( Poess et al., 2014 ) . To build such a decision support system, data engineers can design an ELT pipeline: first, extracting and loading data into the data warehouse, followed by writing transformation queries to process the data for analysis. Compared to traditional Extract-Transform-Load (ETL) pipelines, ELT pipelines ingest data directly into data warehouses, enabling real-time Business Intelligence (BI) analysis ( Waas et al., 2013 ) . Furthermore, with cloud infrastructure, ELT enhances scalability for processing large volumes of data ( Simitsis et al., 2023 ) and offers greater flexibility in incorporating additional data transformations ( Poppy, 2023 ) . These benefits make ELT pipelines an increasingly preferred choice for processing data across various scenarios ( Seenivasan, 2022 ; Mbata et al., 2024 ; Foidl et al., 2024 ; Singhal and Aggarwal, 2022 ; Simitsis et al., 2023 ) . Figure 1 . ELT-Bench is the first end-to-end benchmark designed to evaluate the ability of AI agents to build ELT pipelines. The agent must construct complete ELT pipelines from scratch by decomposing the complex workflow, interacting with databases and data tools, writing code and SQL queries, and calling APIs. Developing ELT pipelines is an essential task for data engineers ( Seenivasan, 2022 ; Singhal and Aggarwal, 2022 ; Mbata et al., 2024 ; Foidl et al., 2024 ; Simitsis et al., 2023 ) , but the process requires significant manual work. Prior studies estimate that data engineers spend over 60% of their time on data warehousing projects building data pipelines ( Rasool and Malik, 2015 ; Kimball and Caserta, 2004 ; Widom, 1995 ; Hernández and Stolfo, 1998 ; Chatterjee and Segev, 1991 ) . First, these pipelines must extract and integrate data from disparate sources with varying formats and standards. Second, data engineers or analysts need a deep understanding of the source data schema to write transformation queries. Can AI agents effectively reduce the manual effort involved in constructing ELT pipelines? Recent advancements in Large Language Models (LLMs) have demonstrated strong capabilities in the text-to-SQL task, a crucial component of ELT pipelines. Notably, state-of-the-art (SOTA) techniques based on LLMs have achieved execution accuracy rates of 77.1% and 91.2% on the Bird ( Li et al., 2024a ) and Spider 1.0 ( Yu et al., 2019 ) benchmarks, respectively. Researchers have recently developed AI agents to tackle more complex real-world tasks that demand reasoning, tool usage, planning, and memorization ( Yao et al., 2023b ; Shinn et al., 2023a ; Yang et al., 2024 ; Lei et al., 2024 ; Wang et al., 2024 ; Weng, 2023 ) . To evaluate the capability of emerging AI agents, researchers have proposed numerous benchmarks in the data domain ( Lai et al., 2022 ; Huang et al., 2024a ; Hu et al., 2024 ; Cao et al., 2024 ; Lei et al., 2024 ) . However, there is no end-to-end benchmark designed with end-to-end ELT pipelines. Building an end-to-end ELT benchmark is challenging because it requires sophisticated setup and configuration, time-consuming ground truth labeling, and thorough workflow verification to ensure reproducibility and correctness. First, annotators must set up various data management systems and platforms to store source data and provide data tools capable of handling diverse data sources. Second, annotators must prepare all necessary input files within the project base. Third, annotators must label ground truth by developing configuration files and writing complex transformation queries involving various relational operations (e.g., casting, type conversion, joins, aggregation, and ranking). Finally, annotators manually execute and verify each ELT pipeline to validate the correctness of both the configured environment and generated annotations. We address these challenges by introducing ELT-Bench, a new benchmark of 100 ELT pipelines associated with 835 source tables and 203 data models across various domains. For a single ELT pipeline, we spend approximately 3 to 5 hours of manual effort setting up the environment, annotating input files and the ground truth, and building the entire pipeline for verification. Notably, 60% of the pipelines require extracting and integrating data from five distinct categories of sources (APIs, cloud services, relational databases, NoSQL databases, and flat files). In addition, the ground truth for each pipeline, on average, involves 187 lines of code per configuration file and 200 SQL tokens (tokenized by whitespace ( Lei et al., 2024 ) ) per data model. ELT-Bench is the first benchmark that covers the entire workflow for building ELT pipelines, providing a comprehensive evaluation through several interconnected subtasks. As shown in Figure 1 , ELT-Bench requires agents to construct an end-to-end ELT pipeline from scratch, encompassing two primary stages: (1) data extraction & loading stage and (2) data transformation stage. Agents execute their actions within a sandbox environment, which includes preinstalled packages and an established project base. This setup replicates the real-world workflow of a data engineer, challenging AI agents to break down the complex workflow into manageable subtasks, interact