AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark
Lan Li, Liri Fang, Bertram Ludäscher, Vetle I. Torvik · 2024
Read AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark by Lan Li, Liri Fang, Bertram Ludäscher, Vetle I. Torvik (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06724.txt.
Introduction
Cleaning a messy spreadsheet by hand is slow and slip-prone, even in good tools. Lan Li, Liri Fang, Bertram Ludäscher, and Vetle Torvik built a system in 2024 that hands the job to a language model. Given a raw table and the purpose the clean table must serve, it writes a sequence of cleaning steps in a popular tool's language. The steps cover six common flaws, from bad formats to duplicates.
To judge it they built a benchmark of 142 purposes over ninety-six tables, scoring the final answer, the table itself, and how human-like the steps are. Three open models all lifted data quality over the baseline; the largest wrote the best tables, its smaller sibling the most human-like workflows.
Catalog record
| CREATOR | Lan Li, Liri Fang, Bertram Ludäscher, Vetle I. Torvik |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark Lan Li, Liri Fang, Bertram Ludäscher, Vetle I. Torvik Abstract Data cleaning is a time-consuming and error-prone manual process even with modern workflow tools like OpenRefine. Here, we present AutoDCWorkflow, an LLM-based pipeline for automatically generating data-cleaning workflows. The pipeline takes a raw table coupled with a data analysis purpose, and generates a sequence of OpenRefine operations designed to produce a minimal, clean table sufficient to address the purpose. Six operations correspond to common data quality issues including format inconsistencies, type errors, and duplicates. To evaluate AutoDCWorkflow, we create a benchmark with metrics assessing answers, data, and workflow quality for 142 purposes using 96 tables across six topics. The evaluation covers three key dimensions: (1) Purpose Answer : can the cleaned table produce a correct answer? (2) Column (Value) : how closely does it match the ground truth table? (3) Workflow (Operations) : to what extent does the generated workflow resemble the human-curated ground truth? Experiments show that Llama 3.1, Mistral, and Gemma 2 significantly enhance data quality, outperforming the baseline across all metrics. Gemma 2-27B consistently generates high-quality tables and answers, while Gemma 2-9B excels in producing workflows that resemble human-annotated versions. 1 Introduction Figure 1: In a purpose-driven data cleaning model, AutoDCWorkflow leverages LLM agents to decide each reasoning step and automatically generate a data cleaning workflow based on a curator-defined analysis purpose and a provided dirty dataset. Data curation and cleaning are critical processes that prepare raw data for analysis, ensuring high quality and reliability Chen et al. (2023) . Implementing a sequence of data operations in the form of a data cleaning workflow improves data quality, enabling downstream analyses and yielding trustworthy results and actionable insights Li et al. (2021) ; Wilkinson et al. (2016) ; Parulian and Ludäscher (2022) ; McPhillips et al. (2019) . To facilitate automation and reuse, tools like OpenRefine Guidry (2020) and Trifacta Kandel et al. (2011) automatically capture reusable, executable workflows that ensure consistent results when re-applied to the same raw input. Data cleaning should ensure data is fit-for-purpose, with the analysis purpose guiding the data quality report to identify relevant data quality issues Wang and Strong (1996) ; Pipino et al. (2002) . These issues determine data cleaning objectives, which define actionable constraints and operations necessary to improve data quality. Consequently, data cleaning objectives drive the design of data cleaning workflows, as illustrated in Figure 1 . Despite advancements, data scientists still spend over 80% of their time on cleaning tasks due to diverse domain requirements Rezig et al. (2019) . Designing effective workflows often involves multiple rounds of selecting appropriate operations and arguments to ensure accuracy Stonebraker et al. (2018) . This process remains time-consuming and error-prone, as it demands domain expertise and a deep understanding of the dataset and its schema, yet it is essential for auditing data analyses and ensuring the quality of the resulting tables. Recent advances in large language models (LLMs) have sparked interest in their application to data cleaning Eltabakh et al. (2024) ; Biester et al. (2024) ; Qi and Wang (2024) . However, few existing approaches effectively automate the design of executable data cleaning workflows that are aligned with specific analytical purposes and capable of addressing diverse data quality issues, despite the availability of modern cleaning tools (see Table 1 ). To address this gap, we propose AutoDCWorkflow pipeline and dataset that jointly fulfill three critical criteria: (1) is guided by a data analysis purpose, (2) supports the resolution of multiple data quality issues, and 3) generates executable and reusable data cleaning workflows. Frameworks Task Workflow Multi-DQ Purpose NADEEF Dallachiesa et al. (2013) Rule-based DC ✓ ✓ ActiveClean Krishnan et al. (2016) DC for ML ✓ Holoclean Rekatsinas et al. (2017) ML for DC ✓ SAGA Siddiqi et al. (2023) DC for ML ✓ ✓ ✓ * RetClean Eltabakh et al. (2024) LLM for DC ✓ LLMClean Biester et al. (2024) LLM for DC ✓ CleanAgent Qi and Wang (2024) LLM for DC ✓ COMET Mohammed et al. (2025) DC for ML ✓ ✓ AutoDCWorkflow LLM for DC ✓ ✓ ✓ Table 1: Comparison of existing data cleaning (DC) frameworks and AutoDCWorkflow in terms of whether the framework (1) generates an executable data cleaning workflow ( Workflow ), (2) supports multiple data quality issues ( Multi-DQ ), and (3) is guided by a specific data analysis purpose ( Purpose ). * refers to the purpose being defined by downstream machine learning performance, rather than a data analysis purpose. Two natural questions arise in the context of LLM-assisted data cleaning: (1) Can LLMs help users design efficient, purpose-driven data cleaning workflows? (2) How can we evaluate LLMs’ ability to automate these workflows? Therefore, we propose (1) an LLM-based pipeline, AutoDCWorkflow, that Auto matically generates a D ata C leaning Workflow given a data analysis purpose, and (2) a new dataset benchmark for evaluating automated workflow generation. AutoDCWorkflow pipeline leverages LLMs as data cleaning agents to assess data quality and generate a sequence of operations based on the analysis purpose and raw table, inspired by the purpose-driven data cleaning model Li and Ludäscher (2023) . It features three iterative, prompt-based components, as shown in Figure 2 , and continues iterating until no data quality issues remain. Notably, AutoDCWorkflow is a companion tool built on OpenRefine. While the pipeline itself is programming language-agnostic, we use OpenRefine operations for demonstration in our experiments. The benchmark evaluates LLM agents’ ability to automate data cleaning workflows. AutoDCWorkflow provides annotated datasets that include purposes, answers, raw tables, manually curated clean tables, and workflows. The raw tables are sampled from six real-world datasets in domains like social science, public health, finance, and transportation. Figure 2: Architecture of the AutoDCWorkflow framework. Given a data analysis purpose \mathbf{P} : Identify the main facility types that have been inspected at the highest risk level, and a dirty table \mathbf{T} , we first run the purpose on \mathbf{T} and obtain an incorrect answer \mathbf{A} : “SCHOOOL, RESTUARANT, school, GROCRY STORE” . An iterative data cleaning process is then initiated, consisting of: (1) Selecting the Target Column , (2) Inspecting Column Quality , (3) Generating Operations & Arguments , and (4) Executing Operations via the OpenRefine API. In each iteration, the framework predicts and applies the next operation, evaluates the resulting column quality, and continues until all target columns satisfy the required quality standards (i.e., all dimensions are marked as True in the Data Quality Report). This process yields a cleaned table \mathbf{T}_{N} and a corresponding complete workflow \mathbf{W}_{N} . Running the purpose on \mathbf{T}_{N} produces the correct answer \mathbf{A} : “SCHOOL, RESTAURANT, GROCERY STORE”. Summary of Contributions. AutoDCWorkflow introduces a fully automated LLM-driven framework for data cleaning workflow generation. Our benchmark and evaluation metrics establish a foundation for future advancements in LLM-powered data cleaning. Our contributions include: (1). We design a three-stage LLM-based pipeline—Select Target Columns, Inspect Column Quality, and Generate Operations & Arguments—for automated data cleaning workflow generation. (2). We construct a dataset benchmark to evaluate data cleaning workflows across three dimensions: Purpose Answer, Column Value, and Workflow (Operations). (3). We assess the performance of state-of-the-art models including Llama 3.1–8B, Mistral–7B, Gemma 2–9B and