Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines

2025

Read Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15874.txt.

Introduction

Yuhang Ge and co-authors define a new task: turning plain language requests into data cleaning pipelines, in work from 2025. Preparing raw data for use takes skill and time. Non-experts often know what they want but cannot code the steps.

The team built a large test set from real production patterns applied to over twenty-three thousand tables. The set holds about eighteen thousand tasks across sixteen core operations. Top language models struggled with both multi-step logic and exact settings. A baseline agent that checks its own work step by step did best, yet a wide gap remained. The open test gives the field a shared yardstick for progress.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines Abstract Data preparation (DP) transforms raw data into a form suitable for downstream applications, typically by composing operations into executable pipelines. Building such pipelines is time-consuming and requires sophisticated programming skills. If we can build the pipelines with natural language (NL), the technical barrier of DP will be significantly reduced. However, constructing DP pipelines from NL instructions remains underexplored. To fill the gap, we introduce Text-to-Pipeline , a new task that translates NL data preparation instructions into DP pipelines. Furthermore, we develop a benchmark named PARROT to support systematic evaluation. To simulate realistic DP scenarios, we mined transformation patterns from production pipelines and instantiated them on 23,009 real-world tables collected from six public sources. The resulting benchmark comprises ~18,000 pipelines covering 16 core DP operators. We evaluated cutting-edge large language models on PARROT and observed that they only solved 72.86% of the cases, revealing notable limitations in instruction understanding and multi-step reasoning. To address this, we propose Pipeline-Agent , a stronger baseline that iteratively predicts and executes operations with intermediate table feedback, achieving the best performance of 76.17%. Despite this improvement, there remains substantial room for progress on Text-to-Pipeline . Our data, codes, and evaluation tools are available at https://anonymous.4open.science/r/Text-to-Pipeline . 1 Introduction Data preparation (DP) refers to the process of transforming raw data into a form suitable for downstream applications such as business intelligence (BI) and machine learning (ML) chai2022data ; fernandez2023large ; zha2025data ; zhang2023jellyfish . As a core component of modern data management, tabular DP plays a central role in supporting workflows in data warehouses and BI systems chai2022data . Preparing tabular data typically involves multiple operations such as filtering naeem2024retclean , joining deng2024lakebench , grouping auto-suggest , and reshaping he2018transform . These operations are often composed into pipelines where each step incrementally transforms the table and feeds the result into the next auto-pipeline ; lai2025autoprep ; auto-suggest . However, building correct and efficient pipelines is time-consuming and requires sophisticated programming skills, which is challenging even for experienced data engineers, as they need to compose pipelines in a large compositional space. Besides, this poses a significant technical barrier for non-experts and prevents them from participating in DP. To lower this barrier, a natural language (NL) interface that allows users to complete DP tasks by writing NL instructions is a natural choice. This paradigm has been widely explored in tasks such as Text-to-SQL yu2018spider ; lei2024spider , spreadsheet formula generation maspreadsheetbench ; zhao2024nl2formula , and code generation wei2024selfcodealign ; zhang2024codeagent . However, these tasks primarily focus on database queries, cell-wise functions, or logic expressions, and are not designed to support DP pipeline generation. In parallel, several methods have explored automated pipeline construction auto-pipeline ; lai2025autoprep ; auto-suggest , which often relies on structured supervision, such as input-output table pairs he2018transform ; autopandas ; auto-pipeline or schema graphs lai2025autoprep , making them inapplicable in scenarios where only NL instructions are available. Figure 1 : Task overview of Text-to-Pipeline . To bridge the gap, we introduce Text-to-Pipeline task: translating NL instructions into executable DP pipelines over tabular data. We formalize it as symbolic program generation in a domain-specific language (DSL), which can be compiled into executable backend code such as Pandas or SQL. DSL offers a structured representation, backend flexibility, and stronger support for verification and evaluation than directly generating code (see Sec. 3 ). As shown in Fig. 1 , given the instruction “ Summarize sales by region, and select the top-5 products. ” and input tables, the system generates a pipeline in DSL, e.g., GroupBy , Topk , Select , which is compiled into backend code and executed to produce the final output table. To support this task, we introduce PARROT , a large-scale benchmark for Text-to-Pipeline , which contains ~18,000 multi-step DP tasks spanning 16 core operators, built on 23,009 real tables from six public data sources. Constructing such a benchmark poses several challenges: First, pipelines must be valid and faithfully aligned with NL instructions. Second, NL should capture diverse and realistic intent. Third, operator sequences must be grounded in real table semantics. To address these challenges, we design a five-stage synthesis framework. Specifically, we begin by curating tables from diverse domains and extracting transformation patterns from production pipelines, ensuring that pipelines align with practical usage. Next, we sample DSL programs via Markov processes guided by empirically derived transition matrices, capturing realistic compositional structures. We then generate intent-aligned NL instructions using LLMs and refine them to reflect natural, human-like expression. Each DSL program is compiled into executable backend code, enabling automated evaluation. Finally, we perform multi-phase validation involving six PhD-level human experts to ensure consistency between instructions and programs and verify correctness through execution results. To evaluate the Text-to-Pipeline task and the PARROT benchmark, we compare three settings: zero-shot LLM prompting, structured generation, and agent-based planning. Zero-shot prompting with GPT-4o yields high program validity ( 81.12% ) but limited execution accuracy ( 71% ). Fine-tuned models like Qwen2.5-Coder-7B improve execution accuracy to 74.15% , showing the benefit of high-quality supervised data. Among structured generation, symbolic DSLs achieve the best execution accuracy ( 57.48% ), outperforming Pandas ( 33.8% ) and SQL ( 3.05% ), highlighting the advantage of modular, schema-aware DSL formalism. To explore the planning capabilities of LLM agents, we propose Pipeline-Agent , which follows an iterative predict-execute-reflect cycle and uses intermediate table states for contextual reasoning, achieving the best performance of 76.17% . Nevertheless, there remains substantial room for progress on PARROT . Current LLM-based methods still struggle with instruction grounding , compositional reasoning , and dynamic schema evolution , especially in longer or more complex pipelines. Beyond serving as a benchmark for Text-to-Pipeline , PARROT provides a testbed for exploring LLM-based tool use in data engineering. Combined with the Model Context Protocol (MCP) mcp , which standardizes how LLMs interact with tools and data sources, it lays the foundation for scalable and autonomous data management over large-scale data lakes. 2 Related Work NL-Driven Program Generation. Prior research has extensively studied how to translate NL into executable programs. Specifically, Text-to-SQL methods yu2018spider ; lei2024spider ; li2023bird ; li2024codes map NL queries to SQL statements, emphasizing semantic understanding and schema linking. These approaches are typically benchmarked on datasets such as WikiSQL zhong2017seq2sql , Spider yu2018spider , Spider 2.0 lei2024spider , and BIRD li2023bird . Text-to-Formula zhao2024nl2formula ; li2023sheetcopilot ; maspreadsheetbench techniques like NL2Formula zhao2024nl2formula , SheetCopilot li2023sheetcopilot , and SpreadsheetBench maspreadsheetbench focus on converting NL into spreadsheet formulas or targeted cell-level edits, suitable for localized spreadsheet manipulations. Additionally, general Text-to-Code frameworks feng2020codebert ; chen2021evaluating ; zhang2023survey ; hendrycks2021measuring translate NL instructions into scripts in languages like Python, C++, and Java, addressing diverse standalone programming tasks tested on HumanEval chen2021evaluating and APPS hendrycks2021measuring . Although these paradigms leverage NL, these tasks typically target the level of SQL queries, cell-wise formulas for spreadsheets, or logic functions for general programming. In contrast, Text-to-Pipeline focuses on generating multi-step, executable pipelines for data preparation, where the objective is to transform input tables into expected outputs through schema-aware DSL programs. Automatic Data Pipeline Generation. Automating data pipeline construction is often framed as program synthesis. Early methods rely on manual coding or visual tools trifacta

Read the complete text