Efficient Model Repository for Entity Resolution: Construction, Search, and Integration
Victor Christen, Peter Christen · 2024
Read Efficient Model Repository for Entity Resolution: Construction, Search, and Integration by Victor Christen, Peter Christen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09355.txt.
Introduction
Victor Christen and Peter Christen build a model library for record matching in a paper from twenty twenty-four. Entity resolution decides whether records from different sources describe the same thing. It is a basic step in data integration. With many sources, heterogeneity and scale make pair classification hard.
Existing multi-source methods need labeled record pairs for each task. They also fail to reuse models across related tasks. The authors propose MoRER, a repository of classification models for resolution problems. Feature distribution analysis clusters similar tasks together. That clustering lets the repository start new tasks with moderate labeling effort. Experiments on three multi-source data sets matched or beat existing methods. Readers get a way to stop retraining matchers from scratch.
Catalog record
| CREATOR | Victor Christen, Peter Christen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-09355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Model Repository for Entity Resolution: Construction, Search, and Integration Victor Christen, Peter Christen Abstract . er ( er ) is a fundamental task in data integration that enables insights from heterogeneous data sources. The primary challenge of ER lies in classifying record pairs as matches or non-matches, which in multi-source er (MS-ER) scenarios can become complicated due to data source heterogeneity and scalability issues. Existing methods for MS-ER generally require labeled record pairs, and such methods fail to effectively reuse models across multiple er tasks. We propose MoRER ( Model Repositories for Entity Resolution ), a novel method for building a model repository consisting of classification models that solve er problems. By leveraging feature distribution analysis, MoRER clusters similar er tasks, thereby enabling the effective initialization of a model repository with a moderate labeling effort. Experimental results on three multi-source datasets demonstrate that MoRER achieves comparable or better results to methods that have label-limited budgets, such as active learning and transfer learning approaches, while outperforming self-supervised approaches that utilize large pre-trained language models. When compared to supervised transformer-based methods, MoRER achieves comparable or better results, depending on the size of the training data set used. ER entity resolution AL active learning ML machine learning LLM large language model DOI: XXXXXXX.XXXXXXX Conference: 29th International Conference on Extending Database Technology (EDBT); 24th March-27th March, 2026; Tampere, Finland 1. Introduction Figure 1 . Motivation of reusing solved er tasks for new tasks. The data sources D_{1} and D_{2} are already linked utilizing similarity feature vectors and a model M_{1,2} to label each record pair. The question is whether the derived model M_{1,2} can also be applied to the new data source, D_{3} , to match D_{3} to D_{1} and D_{2} , or if new models have to be generated. Due to the increasing number of available data sources, data integration is essential for comprehensive analysis and a standardized view of information that is often scattered across sources ( Stonebraker and Ilyas, 2018 ) . A cornerstone of data integration is er ( er ), which identifies and links records that refer to the same real-world entity ( Christen, 2012 ) . The broad range of potential data quality issues and data source heterogeneity make er challenging ( Christen, 2012 ; Christophides et al., 2021 ) . Multi-source er ( Christophides et al., 2021 ) , especially, amplifies multiple challenges that range from scalability issues to achieving high-quality linkage results. The quality of the er result between two data sources is highly dependent on the classification step ( Stonebraker and Ilyas, 2018 ) . This step classifies record pairs into matches and non-matches based on attribute value similarities or derived embeddings from pre-trained language models. State-of-the-art methods use machine learning (ML) based approaches since they have been shown to achieve high-quality results ( Doan et al., 2020 ) . However, the majority of ML methods require labeled training data to build a classification model, where generating such data can be a time-consuming and expensive task ( Mudgal et al., 2018 ) . Various methods have been proposed to reduce the labeling effort, such as al ( al ) ( Christen et al., 2019 ; Ngonga Ngomo and Lyko, 2012 ; Mozafari et al., 2014 ; Primpeli and Bizer, 2021 ) and transfer learning ( Kirielle et al., 2022 ; Jin et al., 2021 ; Loster et al., 2021 ) . However, to the best of our knowledge, no existing method supports the reuse of trained er classifiers by building and utilizing a model repository for er . Typical use cases for model repositories are found in research institutions and companies that require the continuous integration of new and diverse data sources. For example, in healthcare, hospitals and clinics often need to integrate patient records from various systems to create unified medical histories in order to enable better diagnosis, treatment, and research ( Padmanabhan et al., 2019 ) . Similarly, in census data analysis, government agencies must reconcile records from multiple surveys and administrative sources to ensure accurate population statistics and policy planning ( Linacre et al., 2022 ) . In e-commerce, product comparison portals and marketplaces integrate product catalogs from different vendors to provide search and recommendation functionalities for users. In all these scenarios, our method will allow organizations to reuse previously trained models from solved er tasks, significantly reducing both labeling efforts and computational costs. Furthermore, our method lays the groundwork for establishing er matching services, enabling users to solve any er problem by leveraging existing models. This capability is particularly valuable in dynamic environments where new data sources are continuously added, ensuring scalability and efficiency in real-world applications. Fig. 1 illustrates the motivation of our proposed method. Assuming we have two integrated data sources, D_{1} and D_{2} , consisting of products specified by the shown attributes. The integration of these data sources requires the solution of the er problem given by the classification task of the similarity feature vectors into matches and non-matches. These similarity features are derived from the attribute-wise comparisons between record pairs. Therefore, the classification model, M_{1,2} , based on labeled record pairs, is trained and used to classify the record pairs A1-B1,A2-B2,A3-B3,A3-B4\text{ and }A4-B1 . Later on, a new data source D_{3} needs to be integrated, where we calculate similarity features between data sources D_{1} and D_{3} , as well as between D_{2} and D_{3} . In a naive solution, we would build one new classification model, M_{1,3} and M_{2,3} , for each new er task between the data sources D_{1} and D_{3} and D_{2} and D_{3} , respectively. Due to the increasing number of models with the growing number of integrated data sources, the labeling effort for generating training data would, however, increase substantially. Instead of building the new models M_{1,3} and M_{2,3} , our aim is to reuse the existing model M_{1,2} that has already solved the (potentially similar) er task between D_{1} and D_{2} . As the amount of data to be integrated increases, it becomes crucial to efficiently generate training data for similar ER problems to ensure effective and scalable classification. Our proposed method, MoRER , addresses these challenges. An alternative approach would be to unify all existing er problems to create a comprehensive training dataset. However, individual er problems can have different distributions of their similarity features. This would make it challenging for a single classification model to account for the potentially diverse similarity distributions. To overcome this issue, the idea would be to create new training data for each new er problem and combine these with existing one. However, such an approach is not scalable, since it requires the training data generation for each new ER problem. Fig. 2 illustrates the variations in similarity distributions calculated by the Jaccard string comparison function ( Christen, 2012 ) on the title for products on the WDC computer dataset ( Primpeli et al., 2019 ) , which consists of four data sources. Given these very diverse similarity distributions, a single unified classification model would struggle to capture the diverse characteristics of matches and non-matches. Moreover, new er problems might not fit into an existing model, so retraining is required. To the best of our