Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolution
Daomin Ji 1, Hui Luo 2, Zhifeng Bao 1, J. Shane Culpepper 3 · 2024
Read Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolution by Daomin Ji 1, Hui Luo 2, Zhifeng Bao 1, J. Shane Culpepper 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00324.txt.
Introduction
Daomin Ji and co-authors tackle table integration in data lakes as three linked tasks, published in 2024. Merging tables into one comprehensive table requires judging whether two rows can join, finding all joinable groups, and settling conflicts among rows. Each step carries its own traps, from typos to meaning differences.
For the first task they train a classifier, and, since labeled data is scarce, a self-supervised method that creates its own training examples, including adversarial ones. Joinable groups then look like dense communities in a graph whose edges are the pairwise judgments, so community detection methods are compared for the second task. The third task resolves conflicts when many rows merge. The authors present the pipeline as an end-to-end treatment of a common data engineering problem.
Catalog record
| CREATOR | Daomin Ji 1, Hui Luo 2, Zhifeng Bao 1, J. Shane Culpepper 3 |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolution Daomin Ji 1, Hui Luo 2, Zhifeng Bao 1, J. Shane Culpepper 3 Abstract Table integration aims to create a comprehensive table by consolidating tuples containing relevant information. In this work, we investigate the challenge of integrating multiple tables from a data lake, focusing on three core tasks: 1) pairwise integrability judgment , which determines whether a tuple pair is integrable, accounting for any occurrences of semantic equivalence or typographical errors; 2) integrable set discovery , which identifies all integrable sets in a table based on pairwise integrability judgments established in the first task; 3) multi-tuple conflict resolution , which resolves conflicts between multiple tuples during integration. To this end, we train a binary classifier to address the task of pairwise integrability judgment. Given the scarcity of labeled data in data lakes, we propose a self-supervised adversarial contrastive learning algorithm to perform classification, which incorporates data augmentation methods and adversarial examples to autonomously generate new training data. Upon the output of pairwise integrability judgment, each integrable set can be considered as a community—a densely connected sub-graph where nodes and edges correspond to tuples in the table and their pairwise integrability, respectively—we proceed to investigate various community detection algorithms to address the integrable set discovery objective. Moving forward to tackle multi-tuple conflict resolution , we introduce an innovative in-context learning methodology. This approach capitalizes on the knowledge embedded within large language models (LLMs) to effectively resolve conflicts that arise when integrating multiple tuples. Notably, our method minimizes the need for annotated data, making it particularly suited for scenarios where labeled datasets are scarce. Since no suitable test collections are available for our tasks, we develop our own benchmarks using two real-word dataset repositories: Real and Join . We conduct extensive experiments on these benchmarks to validate the robustness and applicability of our methodologies in the context of integrating tables within data lakes. ∎ 1 Introduction Data lakes are large repositories that store various types of raw data [ 61 , 37 ] . Recently, there has been a growing interest in performing table discovery tasks [ 42 , 30 , 29 , 95 , 62 , 19 ] to find unionable, joinable or similar tables in large data lakes. The integration of data lake tables into a more unified and comprehensive table can potentially be used to create new knowledge and insights that would otherwise be inaccessible from using the tables in isolation. Specifically, given a set of input tables from data lakes, the objective is to produce a comprehensive table by merging relevant tuples from different tables into unified tuples. To enable table integration in data lakes, four core tasks must be resolved: • Schema alignment. Given two sets of attributes from the tables, the goal is to learn a mapping that aligns each attribute in one set to its corresponding attribute in the other. • Pairwise integrability judgment. For any two tuples, whether from the same table or different ones, determine if they should be integrated. • Integrable set discovery. Based on the judged pairwise integrability, identify all integrable sets across the tables, which indicate exactly which tuples should be integrated together. • Multi-tuple conflict resolution. For each integrable set, produce a single tuple that consolidates all relevant information by reconciling any attribute-level conflicts. Note that pairwise integrability judgment is similar to entity resolution [ 81 , 27 , 44 ] , which aims to determine whether two tuples refer to the same entity. However, in our context, two tuples might be integrated even if they do not strictly correspond to the same entity. For example, a tuple representing a movie and another representing a director may be merged into a new tuple for the director, where the movie becomes an attribute of the director tuple. We also note that existing studies have explored integrating tuples using schema-agnostic entity resolution methods [ 74 , 78 ] , which bypass the schema alignment step. However, in our case, schema-aware methods are preferred and the schema alignment is a prerequisite step, as our objective is to produce a comprehensive table with a unified schema. The aligned schema not only supports this goal but also facilitates subsequent tasks. Example 1 Fig. 1 demonstrates an example of table integration. First, the schemas of all tables must be aligned. While most attributes, such as Movie, Country, and Director, can be directly aligned, the schema alignment method needs to recognize that Actor and Star refer to the same attribute. Next, the tables are combined into an intermediate table T using an outer union operator. During the pairwise integrability judgment phase, it is necessary to evaluate whether any tuple pair can be integrated, even when the tuples contain semantically equivalent values or typographical errors. For example, in tuples t_{1} and t_{5} , “U.S.” and “United States” represent semantically equivalent values, whereas in tuples t_{4} and t_{6} , “United Skates” is a typographical error of “United States.” These cases must be carefully addressed during the pairwise integrability judgment. Based on the pairwise integrability of the tuples, the table T is partitioned into two integrable sets: one related to the movie Titanic and the other to Joker. Finally, the tuples within each integrable set are integrated into a single comprehensive tuple through multi-tuple conflict resolution. Since conflicting values may exist within an attribute in the same integrable set (e.g., “Joaquin Phoenix”and “Tom Cruise” in the Star attribute), the correct value, such as “Joaquin Phoenix,” is selected to produce the final tuple. Figure 1 : A workflow example for data lake table integration Given the high accuracy of existing schema alignment methods, this work focuses on the latter three tasks. Thus, we assume the schemas of all input tables have been aligned by existing schema alignment methods, and the input tables are combined into an intermediate table T . Specifically, integrating tables from data lakes presents the following challenges to these tasks: Pairwise integrability judgment. Compared to relational tables, data lake tables [ 61 ] often contain significant amounts of dirty data, such as typographical errors and missing values (present in the original input tables or introduced through the outer union operator), in addition to semantically equivalent values. As a result, pairwise integrability judgment for data lake tables requires more robust methods to handle these challenges effectively. Furthermore, most existing pairwise integrability judgment methods rely on large amounts of labeled data to train machine learning models [ 27 , 60 ] . However, in the context of data lakes, labeled data is often scarce, making it difficult to apply these methods without addressing the label scarcity issue. Integrable set discovery. Data lake tables may contain massive amounts of dirty tuples that need to be integrated, necessitating a robust and effective approach to accurately identify all integrable sets. Multi-tuple conflict resolution. Existing conflict resolution solutions often rely on truth discovery methods, which estimate the reliability of each data source and select the source with the highest reliability score. However, these methods typically require labeled data, metadata (e.g., citations or page views), or domain knowledge. Such resources are scarce in data lakes, motivating the development