A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution
Sandeepa Kannangara 1 , Arman Abrahamyan 1 , Daniel Elias 1 , Thomas Kilby 2 , Nadav Dar 3 , Luiz Pizzato 1 , Anna Leontjeva 1 , Dan Jermyn 1 · 2025
Read A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution by Sandeepa Kannangara 1 , Arman Abrahamyan 1 , Daniel Elias 1 , Thomas Kilby 2 , Nadav Dar 3 , Luiz Pizzato 1 , Anna Leontjeva 1 , Dan Jermyn 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03767.txt.
Introduction
MERAI, a record-matching pipeline for huge firm data sets, is set out by Sandeepa Kannangara and co-authors in a 2025 arXiv paper. Matching the same person or firm across messy files is called entity resolution. It gets hard at firm scale. MERAI was built to dedupe and link files with millions of rows, and it was checked on real firm work.
Two known tools faced it in tests. One stalled past two million rows for lack of memory. MERAI cleared more than fifteen million rows and kept top match scores in both dedupe and link tasks. The tool suits banks and other large data holders.
Catalog record
| CREATOR | Sandeepa Kannangara 1 , Arman Abrahamyan 1 , Daniel Elias 1 , Thomas Kilby 2 , Nadav Dar 3 , Luiz Pizzato 1 , Anna Leontjeva 1 , Dan Jermyn 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution 2 Affiliated with Commonwealth Bank of Australia at the time of research. Current contact: [email protected] 3 Affiliated with Commonwealth Bank of Australia at the time of research. Current contact: [email protected] Sandeepa Kannangara 1 , Arman Abrahamyan 1 , Daniel Elias 1 , Thomas Kilby 2 , Nadav Dar 3 , Luiz Pizzato 1 , Anna Leontjeva 1 , Dan Jermyn 1 Abstract Entity resolution (ER) remains a significant challenge in data management, especially when dealing with large datasets. This paper introduces MERAI (Massive Entity Resolution using AI), a robust and efficient pipeline designed to address record deduplication and linkage issues in high-volume datasets at an enterprise level. The pipeline’s resilience and accuracy have been validated through various large-scale record deduplication and linkage projects. To evaluate MERAI’s performance, we compared it with two well-known entity resolution libraries, Dedupe and Splink. While Dedupe failed to scale beyond 2 million records due to memory constraints, MERAI successfully processed datasets of up to 15.7 million records and produced accurate results across all experiments. Experimental data demonstrates that MERAI outperforms both baseline systems in terms of matching accuracy, with consistently higher F1 scores in both deduplication and record linkage tasks. MERAI offers a scalable and reliable solution for enterprise-level large-scale entity resolution, ensuring data integrity and consistency in real-world applications. I Introduction Entity resolution (ER) represents a fundamental problem in data management, encompassing the identification of records that refer to identical real-world entities across heterogeneous data sources. Also referred to as entity matching, record linkage [ 1 ] , reference reconciliation [ 2 ] , and merge-purge [ 3 ] , ER addresses the challenge of establishing entity correspondence in the absence of universal identifiers. The significance of this problem has increased substantially with the proliferation of distributed data systems and the exponential growth of enterprise data volumes. Contemporary enterprise environments present particularly challenging conditions for ER implementation. Organisations typically maintain multiple data systems across business units, often including legacy applications with heterogeneous data schemas and inconsistent entity representations. The Commonwealth Bank of Australia (CBA), managing data for approximately 17 million customers [ 4 ] , exemplifies these challenges at scale. The institution’s data infrastructure encompasses numerous digital platforms, necessitating sophisticated ER capabilities to maintain data consistency and support analytical processes. The enterprise ER problem is characterised by several critical requirements that distinguish it from traditional academic formulations. Processing datasets containing tens of millions of records requires computational complexity that scales linearly rather than quadratically with dataset size. Enterprise applications demand precision and recall levels suitable for high-stakes environments where entity misidentification carries significant operational and regulatory consequences. These systems must complete large-scale deduplication and linkage operations within operationally acceptable time constraints. Despite extensive academic research in ER [ 5 ] , current solutions remain inadequate for practical enterprise deployment. According to Barlaug [ 6 ] , most industrial deduplication and record linkage continues to be performed ad-hoc on a case-by-case basis, failing to leverage published ER research. They argue that this gap exists primarily due to limited production readiness, as openly accessible, well-documented, production-ready libraries developed from high-quality ER research are rare. The authors further observe that organisations exhibit reluctance to invest significantly in developing and integrating ER solutions due to increased failure risk. Additional challenges arise from scalability deficiencies, where most available solutions are not designed to perform ER on large datasets, exhibiting poor performance when scaled to millions or billions of records. This limitation becomes more pronounced as datasets grow increasingly complex and heterogeneous. Current ER systems also suffer from incomplete pipeline coverage, typically supporting only specific pipeline components such as indexing and matching [ 7 ] , while neglecting critical stages such as clustering. These empirical findings demonstrate a substantial gap between academic ER research and practical enterprise requirements, motivating the development of MERAI (Massive Entity Resolution using AI). MERAI addresses these limitations through novel algorithmic optimisations within an architecture designed to satisfy enterprise-scale demands for accuracy, scalability, and reliability. Following its implementation, MERAI was widely adopted across the bank for executing ER projects. This research contributes to the field through systematic characterisation of enterprise-scale ER requirements and empirical analysis of existing solution limitations. The work presents improved algorithms through the development of optimised blocking and clustering techniques that achieve linear computational complexity while maintaining high accuracy. The paper describes the design and implementation of a complete ER pipeline encompassing the full workflow from data preprocessing through indexing, matching, and clustering stages, capable of processing datasets at enterprise scale. Additionally, detailed empirical evaluation demonstrates strong performance compared to state-of-the-art frameworks including Dedupe [ 8 ] and Splink [ 9 ] across multiple evaluation metrics. The paper is structured as follows: Section II provides technical background on ER methodology and reviews existing solutions. Section III presents the MERAI system architecture and detailed algorithmic specifications. Section IV reports experimental results from comparative evaluations with established ER frameworks across datasets of varying scale and complexity. II Entity Resolution ER encompasses the systematic identification of records that refer to identical real-world entities within or across data sources when unique identifiers are unavailable. The ER process operates on structured data records, addressing both deduplication (within single datasets) and record linkage (across multiple datasets) [ 10 ] . The standard ER workflow comprises three sequential stages, as illustrated in Figure 1 [ 11 , 5 ] : Entity descriptions Indexing Entity Matching Clustering Grouped entity descriptions Entity Resolution Fig. 1 : Entity Resolution Workflow II-A ER Workflow Components II-A 1 Indexing The technique of indexing, also known as blocking, tackles the computational burden created by exhaustive pairwise comparisons. For datasets of size n , naive comparison requires (n^{2}-n)/2 operations, resulting in quadratic complexity [ 5 ] . Indexing techniques group potentially matching records into blocks based on shared characteristics, ensuring comparisons occur only between candidates within common blocks [ 12 ] . This approach considerably reduces computational requirements while maintaining recall through careful selection of blocking criteria. II-A 2 Entity Matching The matching stage determines whether candidate pairs identified during indexing refer to identical entities. Contemporary approaches employ various methodologies including string similarity measures, probabilistic models, supervised learning, unsupervised learning, and rule-based systems [ 13 ] . Machine learning-based techniques demonstrate superior accuracy by learning matching probabilities from training examples [ 10 ] . Recent research indicates that learning-based approaches consistently outperform rule-based methods in practical applications [ 7 , 14 ] . II-A 3 Entity Clustering Clustering transforms pairwise matching decisions into entity groups representing distinct real-world objects [ 15 , 16 ] . This stage operates on a similarity graph where nodes represent records and weighted edges indicate matching likelihood. Clustering algorithms can identify transitive relationships, grouping entities that lack direct pairwise connections but share common intermediary matches. The output consists of disjoint clusters, each containing all records corresponding to a single real-world entity [ 10 ] . II-B Existing ER Solutions Several open-source and commercial ER tools are available for structured data processing [ 7 , 10 , 17 ] . This section reviews representative solutions that informed our system design decisions: II-B 1 FEBRL The Freely Extensible Biomedical Record Linkage (FEBRL) library [