Multimodal Privacy-Preserving Entity Resolution with Fully Homomorphic Encryption

Susim Roy, Nalini Ratha · 2026

Read Multimodal Privacy-Preserving Entity Resolution with Fully Homomorphic Encryption by Susim Roy, Nalini Ratha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18612.txt.

Introduction

Susim Roy and Nalini Ratha build a way to match records of the same person across files while the facts stay locked. Banks and state desks must often tell if two records name one person. Names shift in form: a nick name here, a typo there, a new home in the next file. The match must be made, yet the facts are too dear to share in plain form.

Their plan works on locked facts with a code that lets maths run while the lock stays on. It takes in more than text, so sight and sound cues can help the match. A locked match could let desks share the gain of a joint check with no leak of names. That would help fight fraud while it keeps faith with strict rules on data.

Catalog record

CREATORSusim Roy, Nalini Ratha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18612

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Privacy-Preserving Entity Resolution with Fully Homomorphic Encryption Susim Roy, Nalini Ratha Abstract The canonical challenge of entity resolution within high-compliance sectors, where secure identity reconciliation is frequently confounded by significant data heterogeneity, including syntactic variations in personal identifiers, is a longstanding and complex problem. To this end, we introduce a novel multimodal framework operating with the voluminous data sets typical of government and financial institutions. Specifically, our methodology is designed to address the tripartite challenge of data volume, matching fidelity, and privacy. Consequently, the underlying plaintext of personally identifiable information remains computationally inaccessible throughout the matching lifecycle, empowering institutions to rigorously satisfy stringent regulatory mandates with cryptographic assurances of client confidentiality while achieving a demonstrably low equal error rate and maintaining computational tractability at scale. † † address: University at Buffalo, The State University of New York Department of Computer Science and Engineering † † footnotetext: © Copyright 2026 IEEE. Published in ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), scheduled for 3-8 May 2026 in Barcelona, Spain. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: + Intl. 908-562-3966. 1 Introduction Table 1: Example dataset entries where Rows 1 and 2 have the same name but different addresses and headshots and Rows 3 and 4 have slight name and address variations with young and aged headshots, respectively. Name Address Image Jonathan Diaz 8708 Harris Road, Popetown, MH 23945 Jonathan Diaz 5190 Thomas Mews, Dicksonburgh, TN 13160 Mrs. Ann Pham 254 Savage River Ste. 120, Miguelfort, GA 9209 Anne Pham 254 Savage River Suite 120, Miguelfort, GA 92091 In critical identity verification processes employed by organizations like the Department of Motor Vehicles(DMVs), passport authorities, and financial institutions, a fundamental challenge is reconciling an individual’s identity over time. While core biometric identifiers like fingerprints and facial structures change minimally, it may not be available because they were not collected since such modalities were not viable for automated matching or unsuitable for biometric matching due to poor quality. The biographical data is often dynamic; individuals frequently relocate, abbreviate their names, or update addresses. This discrepancy creates significant validation hurdles, especially in scenarios where a person has visibly aged while residing far from their last recorded location. This highlights the critical need for automated systems that can holistically verify an identity before issuing official credentials such as passports and driver’s licenses or granting access to regulated services. We identify the main culprit for the aforementioned limitations as relying on a single data source while each source may provide significant identity resolution capacity. A natural extension moves toward multimodal data sources. This synergy of combining different data types can produce more reliable and comprehensive authentication outcomes. Existing methodologies [ 13 , 2 , 15 , 3 ] , while addressing multimodal identity traits for entity resolution, are constrained by limitations such as dependence on unaligned and restricted Pinellas County Sheriff’s Office(PCSO) dataset, susceptibility to the inherent recognizability of celebrity faces by Vision-Language Models [ 7 ] , susceptibility to uncontrolled headshot variations, or reliance on highly specific biographic data. Additionally, utilizing multiple biometric signatures also elevates the security risks. Biometric templates are not just random data; they contain significant information that could potentially be used to reconstruct a person’s identity or leak other soft-biometric details. Therefore, it is imperative that any fusion technique is not only robust but also preserves the privacy of the individual’s data. To address this critical need, researchers have turned to advanced cryptographic methods. Homomorphic Encryption (HE), for instance, allows for computations to be performed directly on encrypted data. This enables the fusion of biometric templates without ever decrypting them, thus keeping the sensitive information secure throughout the process [ 12 ] . This principle extends to the fusion of classifier outputs in ensemble learning systems, where Fully Homomorphic Encryption (FHE) can be used to build privacy-preserving algorithms that securely process data while it remains encrypted, ensuring both high performance and stringent data security [ 11 ] . In contrast to the previous work, our contribution is as follows: (1) Entity Resolution is processed in an encrypted domain preserving the user’s privacy (2) Multimodal learning stream using biometric and biographic data (3) We release a novel synthetic dataset simulating a real-world entity resolution scenario. 2 Methodology Figure 1: The above figure demonstrates the training pipeline to obtain the biometric and biographic score vectors. Here, s^{1}_{bm} and s^{1}_{bg} denote the score vectors for one unique individual. 2.1 Dataset and Pipeline Overview Figure 2: The above figure demonstrates the encrypted enrollment and authentication pipeline. The three modality databases store the encrypted biometric, fused, and biographic templates, which are then matched during verification. Table 2: The table below shows the summary of our multimodal dataset creation and statistics. The alternate address spellings and common name variations were created with Claude 3.5-Sonnet [ 1 ] based on the LM-Arena rankings. Stage Description # Individuals # Records # Images Data Augmentation Steps Initial Base unique individuals 36,661 36,661 36,661 Step 1 Added alternate address spellings 36,661 46,365 46,365 Step 2 Added 4-7 random new addresses per person 36,661 266,098 266,098 Step 3 Added common name variations with 2-7 addresses 36,661 287,532 287,532 Final Dataset Split Training Set Used for model training 21,661 \sim 277,532 \sim 277,532 Testing Set Used for evaluation 10,000 10,000 10,000 - Enrolled Individuals registered in the test database 5,000 5,000 5,000 - Verification Individuals used for test queries 5,000 5,000 5,000 Total Final Generated Dataset 36,661 287,532 287,532 We build on top of [ 8 ] by selecting the \langle\text{Name},\text{Address}\rangle attributes out of 8 attributes related to the financial information of different synthetically generated people. Now let us consider the following scenarios where an individual’s biographic data maybe altered (1) An individual can write their name or address in different formats either intentionally or mistakenly, e.g., Anne , Street and Boulevard can be written as Ann , St. and Blvd respectively(2) An individual may have lived at different addresses overlapping with another individuals’s addresses (3) Each individual’s biographic data might be related to different headshot(biometric) images taken over different time periods, e.g., a person might look aged than when they first did their Know Your Customer(KYC). We used Stable Diffusion XL [ 9 ] to first create a base image on top of which the variegated headshots were produced. Table 2 shows the detailed description of the number of datapoints in our synthetically created dataset after each step and the final set. An example of the data points is shown in Table 1 . On the training set, we first train our model for 5 epochs with a learning rate of 1e-5 and batch size of 256 as shown in Figure 1 . This trained model is used to store 5k entity templates in their respective databases. Once stored, they

Read the complete text