Scalable Unit Harmonization in Medical Informatics Using Bi-directional Transformers and Bayesian-Optimized BM25 and Sentence Embedding Retrieval

Jordi de la Torre · 2025

Read Scalable Unit Harmonization in Medical Informatics Using Bi-directional Transformers and Bayesian-Optimized BM25 and Sentence Embedding Retrieval by Jordi de la Torre (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00810.txt.

Introduction

Jordi de la Torre presents a 2025 system that fixes messy lab units in huge health data sets. Mismatched units block data sharing across hospitals and studies. His pipeline blends a classic word match score with sentence embeddings. A smart search over settings tunes that blend.

Then a transformer judge re-ranks the top unit picks for each lab entry. Tests ran on a claims data mart holding seven and a half billion entries. On a rank quality gauge, the blend scored zero point eight eight three. Word match alone scored zero point seven nine nine, and embeddings alone came in lower at zero point five two eight. After the judge stepped in, the final score rose to zero point nine eight three. Top pick accuracy hit eighty-three percent, while recall reached near ninety-five percent by rank five.

Catalog record

CREATORJordi de la Torre
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00810

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable Unit Harmonization in Medical Informatics Using Bi-directional Transformers and Bayesian-Optimized BM25 and Sentence Embedding Retrieval Jordi de la Torre Abstract Objective: To develop and evaluate a scalable methodology for harmonizing inconsistent units in large-scale clinical datasets, addressing a key barrier to data interoperability. Materials and Methods: We designed a novel unit harmonization system combining BM25, sentence embeddings, Bayesian optimization, and a bidirectional transformer based binary classifier for retrieving and matching laboratory test entries. The system was evaluated using the Optum Clinformatics Datamart dataset (7.5 billion entries). We implemented a multi-stage pipeline: filtering, identification, harmonization proposal generation, automated re-ranking, and manual validation. Performance was assessed using Mean Reciprocal Rank (MRR) and other standard information retrieval metrics. Results: Our hybrid retrieval approach combining BM25 and sentence embeddings (MRR: 0.8833) significantly outperformed both lexical-only (MRR: 0.7985) and embedding-only (MRR: 0.5277) approaches. The transformer-based reranker further improved performance (absolute MRR improvement: 0.10), bringing the final system MRR to 0.9833. The system achieved 83.39% precision at rank 1 and 94.66% recall at rank 5. Discussion: The hybrid architecture effectively leverages the complementary strengths of lexical and semantic approaches. The reranker addresses cases where initial retrieval components make errors due to complex semantic relationships in medical terminology. Conclusion: Our framework provides an efficient, scalable solution for unit harmonization in clinical datasets, reducing manual effort while improving accuracy. Once harmonized, data can be reused seamlessly in different analyses, ensuring consistency across healthcare systems and enabling more reliable multi-institutional studies and meta-analyses. 1 Background and Significance Modern scientific research increasingly depends on the integration of heterogeneous data sources, with inconsistent units of measurement posing a persistent challenge, particularly in laboratory analyses where identical quantities may be reported using diverse conventions. Data harmonization is a critical prerequisite for large-scale clinical research, ensuring interoperability across electronic health record systems and supporting reliable, reproducible medical analyses. Laboratory unit harmonization is especially complex due to domain-specific terminology and the clinical importance of precise unit conversion. Traditional methods for unit harmonization often rely on manually curated rule-based systems and mapping tables developed by domain experts [ 1 ] . While these approaches can be effective within narrowly defined contexts, they are not easily scalable and require ongoing maintenance to accommodate emerging laboratory codes and units. This labor-intensive process introduces bottlenecks in research workflows, delays analysis, and increases resource burdens. Emerging methodologies that combine machine learning with information retrieval techniques offer promising solutions to these challenges. Techniques such as sentence embeddings, bidirectional transformers, and optimized retrieval models have demonstrated potential for automating and scaling harmonization processes with improved accuracy and efficiency [ 2 , 3 , 4 ] . In this paper, we present a novel automated unit harmonization system that advances current practices by integrating Bayesian-optimized BM25, sentence embeddings, and transformer-based re-ranking mechanisms. Our system directly addresses major limitations in existing frameworks, including poor scalability to billions of records, limited adaptability to diverse naming conventions, inadequate contextual understanding of laboratory codes, lack of dynamic feedback loops, and overreliance on manual validation. By mitigating these constraints, our approach facilitates seamless data integration, reduces error rates, and accelerates the pace of scientific discovery, ultimately empowering clinical researchers to derive more robust, timely, and reproducible insights from increasingly complex and voluminous healthcare datasets. 2 Materials and Methods 2.1 Datasets 2.1.1 Primary Dataset The Optum Clinformatics Data Mart (CDM) constitutes our primary evaluation dataset. This comprehensive, de-identified database integrates administrative health claims data from over 84 million individuals across all 50 U.S. states. The resource contains more than 7.5 billion medical and pharmacy claims, documenting healthcare utilization and associated costs. Data elements include member demographics, detailed medical and pharmacy claims, laboratory results, inpatient confinement records, and provider information. In this study, we used the 2024 Q1, Q2, and Q3 version of the Optum Clinformatics Data Mart. To support the harmonization process, key fields were extracted from the dataset, including LOINC codes, units of measurement, frequency of occurrence, and descriptive statistics such as minimum, maximum, mean, and standard deviation. This process resulted in approximately 30,000 unique triads (test, sample, unit) requiring standardization. Of these, 17,244 entries were identified in our internal database as candidates for matching. 2.1.2 Reference Dataset We utilized an internal Labcodes Standard Database for unit harmonization, providing a comprehensive mapping of laboratory tests with standardized information. The database facilitates consistent data representation across diverse sources by linking original units to preferred units and providing necessary conversion factors. The harmonization process is mapped against multiple fields within this database, including Test Name, Test Label, Synonym, Sample Name, Labcode, Preferred Unit, Actual Unit, Conversion Factor, and various CDISC standardization fields. Although all fields contribute to a comprehensive test definition, harmonization can often be achieved with a minimum of Test Name, Sample Name, and Unit. Inclusion of additional fields improves the accuracy and specificity of the match. 2.2 System Architecture Our harmonization framework implements a multi-stage pipeline comprising data processing, predictive modeling, and validation components. A key design principle is the modular decomposition of this framework into independent blocks, allowing for focused development, rigorous testing, and iterative improvement of individual components. 2.2.1 Overall Pipeline Structure The overall system is structured as an end-to-end processing pipeline (Fig. 1 ) with distinct stages, each designed to address specific challenges in the harmonization process. The pipeline begins with the Download stage , which involves the extraction of data from the relational database management system (RDBMS), focusing on unique combinations of LOINC codes and reported units, accompanied by pre-computed usage statistics. Following this, the Filtering stage removes entries that fall below a predefined frequency threshold. During this phase, the system also enforces unit name normalization and eliminates inconsistencies to ensure data quality and reduce noise in downstream tasks. The Update stage enriches the filtered dataset by incorporating additional contextual information. This is achieved through inner joins using LOINC codes to retrieve descriptive labels and standardized mappings for column names, enhancing interpretability and facilitating accurate retrieval. Next, the Predictive Modeling stage implements the core harmonization intelligence (detailed in Section 2.2.2 ). The pipeline concludes with the Manual Validation stage , which allows domain experts to review and confirm harmonization results through a custom-built user interface. This feedback not only ensures clinical safety and accuracy but also contributes to the iterative improvement of the system. Figure 1: Overview of the harmonization pipeline showing the sequential flow of data from raw database entries to validated harmonized units 2.2.2 Predictive Model Architecture The predictive model architecture (Fig. 2 ) constitutes the core intelligence of our system and consists of three primary components. The first component is the Hybrid Retrieval Engine , which combines two complementary approaches. The Lexical Retrieval Module implements BM25 algorithm-based traditional term-matching for precise keyword identification and excels at exact and fuzzy matching of laboratory terminology. Complementing this, the Semantic Retrieval Module utilizes sentence embeddings to capture contextual relationships between terminology and can identify semantically equivalent terms even when they share limited lexical overlap. The second component is the Bayesian Score Optimizer , which adjusts the weights between lexical and semantic scores to maximize retrieval performance. This component uses a Gaussian Process-based optimization procedure to determine the optimal balance between precision and recall for each query context. Importantly, this optimization is performed

Read the complete text