Prompt2DAG: A Modular Methodology for LLM-Based Data Enrichment Pipeline Generation Journal: Information and Software Technology

Abubakari Alidu, Michele Ciavotta, Flavio De Paoli · 2025

Read Prompt2DAG: A Modular Methodology for LLM-Based Data Enrichment Pipeline Generation Journal: Information and Software Technology by Abubakari Alidu, Michele Ciavotta, Flavio De Paoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13487.txt.

Introduction

Prompt2DAG, a 2025 systems paper by Abubakari Alidu, Michele Ciavotta, and Flavio DePaoli, turns plain-language requests into executable Apache Airflow pipelines, the step-by-step workflows that enrich raw data. Building such pipelines by hand takes deep engineering skill, so the team tests whether large language models can draft them reliably instead.

Four generation styles, direct prompting, model-only, hybrid, and template-based, were compared across many experiments with thirteen language models and five case studies. The hybrid style won, succeeding in just under four fifths of runs and scoring best on reliability, structure, and executability. The authors argue reliability, not raw code quality, separates the methods, and that hybrid generation is the practical path.

Catalog record

CREATORAbubakari Alidu, Michele Ciavotta, Flavio De Paoli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13487

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt2DAG: A Modular Methodology for LLM-Based Data Enrichment Pipeline Generation Journal: Information and Software Technology Abubakari Alidu, Michele Ciavotta, Flavio De Paoli Abstract Developing reliable data enrichment pipelines demands significant engineering expertise. We present Prompt2DAG, a methodology that transforms natural language descriptions into executable Apache Airflow DAGs. We evaluate four generation approaches—Direct, LLM-only, Hybrid, and Template-based—across 260 experiments using thirteen LLMs and five case studies to identify optimal strategies for production-grade automation. Performance is measured using a penalized scoring framework that combines reliability with code quality (SAT), structural integrity (DST), and executability (PCT). The Hybrid approach emerges as the optimal generative method, achieving a 78.5% success rate with robust quality scores (SAT: 6.79, DST: 7.67, PCT: 7.76). This significantly outperforms the LLM-only (66.2% success) and Direct (29.2% success) methods. Our findings show that reliability, not intrinsic code quality, is the primary differentiator. Cost-effectiveness analysis reveals the Hybrid method is over twice as efficient as Direct prompting per successful DAG. We conclude that a structured, hybrid approach is essential for balancing flexibility and reliability in automated workflow generation, offering a viable path to democratize data pipeline development. 1 Introduction Data pipelines are foundational to modern data-driven organizations, enabling the systematic collection, processing, transformation, and analysis of information across diverse domains [ 1 , 2 ] . These pipelines orchestrate workflows that integrate heterogeneous data sources, apply specific business logic, and deliver actionable insights. However, the design, implementation, and maintenance of such pipelines traditionally demand specialized data engineering expertise, creating a significant barrier for domain experts who possess subject-matter knowledge but may lack the specific programming skills needed to translate their requirements into robust, executable workflows [ 3 , 4 ] . This technical barrier restricts broader participation in data pipeline development, thereby limiting the agility with which organizations can innovate and respond to data-driven opportunities. The challenge is particularly evident when using powerful workflow orchestration frameworks that offer flexibility and comprehensive ecosystems, but require careful management of task dependencies, diligent configuration of operators, and often complex debugging processes [ 5 ] . The capabilities demonstrated by Large Language Models (LLMs) in natural language understanding and code generation open up the possibility of bridging this expertise gap by enabling pipeline descriptions in natural language and translating them into structured workflow definitions or executable code [ 6 , 7 ] . However, the efficacy and reliability of LLM-based pipeline generation are heavily contingent on how the generation task is structured and how the LLM’s capabilities are harnessed. Direct, end-to-end generation from high-level descriptions to complex code often struggles with precision, maintainability, and adherence to software engineering best practices [ 8 ] . In this paper, we introduce and evaluate Prompt2DAG , a methodology for transforming natural language descriptions into executable data workflows. Our work focuses specifically on data enrichment pipelines, a critical and broadly applicable category of data processing where organizations enhance their data assets through geocoding, entity reconciliation, or sentiment analysis. We systematically compare four distinct generation strategies—direct, LLM-only, hybrid, and template-based—to identify the architectural patterns that are essential for achieving production-grade reliability. Our primary contribution is the empirical demonstration that a hybrid, template-guided approach provides the optimal balance of flexibility and robustness, significantly outperforming purely generative methods. Unlike end-to-end generation approaches, Prompt2DAG follows a structured, four-stage process-Pipeline Analysis, Workflow Description Generation, Executable Workflow Generation, and Automated Evaluation—to enhance clarity, facilitate validation at each stage, and make workflow creation more manageable and accessible, particularly for users who are not data engineering specialists. Our experiments provide a comprehensive evaluation by leveraging a diverse suite of thirteen state-of-the-art LLMs, including both leading proprietary models like GPT-4o Mini and Claude 3.5 Sonnet, and a wide range of open-source architectures. This ensures our findings are representative of the current state of the art in language model capabilities. The experimental results confirm that a structured, multi-stage approach is critical for reliable DAG generation. The deterministic Template-based method achieved the highest success rate (92.3%), establishing a benchmark for reliability. Among generative methods, our Hybrid approach proved to be the most effective, delivering a robust 78.5% success rate and the highest quality scores across all metrics (SAT: 6.79, DST: 7.67, PCT: 7.76). This represents a significant performance gain over the LLM-only method, which, despite a respectable 66.2% success rate, suffered from lower and more inconsistent quality scores. By contrast, Direct prompting proved unsuitable for practical use, with a low 29.2% success rate and collapsed scores across all quality dimensions. The remainder of this paper is organized as follows. Section 2 defines the experimental framework by specifying the type of pipelines under study, introducing the adopted component model, and outlining the dimensions considered for evaluation. Section 4 describes the case studies addressed in the experiments. In Section 3 , we review prior research on LLM-based code generation and workflow automation. Section 5 presents our Prompt2DAG methodology in detail, covering each stage from pipeline analysis to automated evaluation. Next, Section 6 describes the experimental setup, and discusses our key findings and comparative analyses. Finally, Section 7 summarizes our contributions and outlines promising directions for future research in automated data pipeline development. 2 Background This paper investigates whether, and to what extent, LLMs can be used to generate executable pipelines from natural language descriptions. The objective is to develop a set of best practices and reusable templates to assist domain experts with limited technical expertise in automating their workflows. In this section, we present the context in which the solution was developed, along with the SemT service model assumed in the pipeline design. Finally, the approach to pipeline generation and validation is outlined. 2.1 Problem definition To develop, demonstrate, and evaluate our methodology, we focus on data preparation pipelines as a representative and broadly applicable case study. Data preparation is relevant in many domains, ranging from analytics to training of AI models. In particular, we focus on tabular data enrichment, a critical phase in data preparation that aims to augment or extend existing data with additional information from external sources to enhance its value. A typical tabular data enrichment workflow begins with data cleaning , which corrects errors and inconsistencies to ensure high-quality underlying data [ 9 ] , followed by data formatting , which transforms data to comply with standards and facilitate subsequent processing [ 10 ] . The key task in data enrichment is data reconciliation , which matches source data with data in a shared dataset, typically a knowledge graph, to enable the retrieval of data from another dataset and augment the source table with contextual information. This last task is known as data extension , which involves adding new columns to the source tables to prepare the data for downstream tasks, such as analytics or training AI models. Such workflows typically involve integrating heterogeneous data, applying complex transformations, and managing dependencies between diverse processing steps, making them an ideal testbed. Figure 1 illustrates a practical application where tabular data is enriched with geospatial and meteorological information. The original table (gray) is augmented with geographical coordinates (green), which, combined with date information, enable the retrieval of weather data (orange). This process exemplifies common pipeline operations such as data reconciliation (e.g., city names), data type standardization (e.g., dates), and external

Read the complete text