The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data
Xiaolong Luo, Michael Lingzhi Li · 2025
Read The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data by Xiaolong Luo, Michael Lingzhi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08247.txt.
Introduction
Critical care data from many hospitals could train fairer, more general clinical AI, if the data could be merged cleanly. Xiaolong Luo and Michael Lingzhi Li present CRISP in a 2025 research paper. CRISP is a processing pipeline for the CRITICAL dataset, a multi-institutional collection in a common data model. The dataset holds billions of records from hundreds of thousands of patients at four institutions.
The pipeline manages data quality with full audit trails. It maps varied medical vocabularies onto unified standard codes, deduplicates records, and standardizes units. Its modular design processes the whole dataset in under a day on ordinary hardware. Baseline prediction benchmarks come included, giving future studies a shared starting point. The authors say CRISP saves researchers months of preprocessing. That saved effort can go into the clinical questions instead.
Catalog record
| CREATOR | Xiaolong Luo, Michael Lingzhi Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data Xiaolong Luo, Michael Lingzhi Li Abstract While existing critical care EHR datasets such as MIMIC and eICU have enabled significant advances in clinical AI research, the CRITICAL dataset opens new frontiers by providing extensive scale and diversity—containing 1.95 billion records from 371,365 patients across four geographically diverse CTSA institutions. CRITICAL’s unique strength lies in capturing full-spectrum patient journeys, including pre-ICU, ICU, and post-ICU encounters across both inpatient and outpatient settings. This multi-institutional, longitudinal perspective creates transformative opportunities for developing generalizable predictive models and advancing health equity research. However, the richness of this multi-site resource introduces substantial complexity in data harmonization, with heterogeneous collection practices and diverse vocabulary usage patterns requiring sophisticated preprocessing approaches. We present CRISP (CRITICAL Records Integrated Standardization Pipeline) to unlock the full potential of this valuable resource. CRISP systematically transforms raw Observational Medical Outcomes Partnership Common Data Model data into ML-ready datasets through: (1) transparent data quality management with comprehensive audit trails, (2) cross-vocabulary mapping of heterogeneous medical terminologies to unified SNOMED-CT standards, with deduplication and unit standardization, (3) modular architecture with parallel optimization enabling complete dataset processing in < < 1 day even on standard computing hardware, and (4) comprehensive baseline model benchmarks spanning multiple clinical prediction tasks to establish reproducible performance standards. By providing processing pipeline, baseline implementations, and detailed transformation documentation, CRISP saves researchers months of preprocessing effort and democratizes access to large-scale multi-institutional critical care data, enabling them to focus on advancing clinical AI. The complete source code, baseline models, and documentation are publicly available. keywords Electronic Health Records, CRITICAL Dataset, Intensive Care Unit, Multi-institutional Data, Critical Care, Data Processing Pipeline, Healthcare AI, Real-world Data Data and Code Availability The CRITICAL dataset is available under a data use agreement at https://critical.fsm.northwestern.edu . CRISP source code, documentation, and processing scripts are publicly available at https://github.com/AaronLuo00/CRISP-Pipeline . Institutional Review Board (IRB) This research has been designed by Harvard IRB as Not Human Subject Research. IRB Protocol information will be provided if the paper is accepted. 1 Introduction The rapid advancement of artificial intelligence (AI) has revolutionized healthcare through its integration with electronic health records (EHRs), enabling unprecedented capabilities in clinical prediction, diagnosis, and decision support. Recent breakthroughs have demonstrated the ability to accurately predict multiple medical events from EHR data ( Rajkomar et al. 2018 ; Jiang et al. 2023 ; Grout et al. 2024 ; Hegselmann et al. 2025 ) , with models achieving performance comparable to clinical experts in various domains including mortality prediction, disease diagnosis, and treatment recommendation. These successes have increased interest in developing AI systems that can assist clinicians in real-time decision-making ( Tomasev et al. 2019 ) , predict patient deterioration hours before clinical manifestation ( Lauritsen et al. 2020 ) , and optimize resource allocation in increasingly strained critical care settings where timely intervention can significantly impact patient outcomes ( Komorowski et al. 2018 ; Gutierrez 2020 ) . However, the promise of AI in healthcare critically depends on access to large-scale, diverse, and well-structured clinical data, a fundamental challenge that limit the development of truly generalizable AI models ( Futoma et al. 2020 ; Kelly et al. 2019 ) . Most existing models are trained on single-institution datasets, raising concerns about their transferability across different healthcare systems, patient populations, and clinical practice patterns ( Zech et al. 2018 ; Nestor et al. 2019 ) . Furthermore, the heterogeneity in data collection practices, vocabulary usage, and documentation standards across institutions creates a substantial barrier to developing robust, multi-institutional AI systems that can benefit diverse patient populations ( Gianfrancesco et al. 2018 ; Obermeyer et al. 2019 ) . This challenge is particularly acute in critical care settings, where the complexity of patient conditions, the diversity of monitoring equipment, and the urgency of clinical decisions demand models that can generalize across varying institutional protocols and patient demographics ( Sendak et al. 2020 ; Shah et al. 2019 ) . 1.1 The Data Challenge in Healthcare AI Pioneering datasets like MIMIC-III and MIMIC-IV have established the foundation for critical care AI research, with MIMIC-IV comprising over 65,000 intensive care unit (ICU) patients and more than 200,000 emergency department (ED) patients, enabling groundbreaking advances in mortality prediction, treatment optimization, and clinical decision support ( Johnson et al. 2016 ; Johnson et al. 2023 ) . The eICU Collaborative Research Database further expanded the field by demonstrating the value of multi-center data ( \sim 200,000 ICU admissions across 208 hospitals), pioneering cross-institutional research approaches ( Pollard et al. 2018 ) . These foundational resources have trained a generation of researchers and established methodological standards that continue to guide the field. Building upon these essential contributions, the CRITICAL dataset extends the research landscape by providing 1.95 billion records from 371,365 patients across four Clinical and Translational Science Awards (CTSA) sites, offering complementary strengths including full-spectrum patient journeys (pre-ICU, ICU, and post-ICU), extended longitudinal tracking, and diverse geographic representation ( The CRITICAL Consortium 2025 ) . While this scale and diversity enable more generalizable modeling, they also introduce cross-site semantic heterogeneity—differences in vocabulary usage, coding practices, units, and temporal granularity—demanding transparent, reproducible data preprocessing for cleaning, standardization, and harmonization. CRISP addresses these challenges by providing a modular, reusable pipeline that not only accelerates research but also ensures consistency with the methodological standards established by the MIMIC and eICU communities. 1.2 Our Contributions To address these critical needs for standardization and unified data formats, we present CRISP (CRITICAL Records Integrated Standardization Pipeline), a comprehensive solution that transforms CRITICAL’s 1.95 billion raw records into ML-ready formats. Our contributions include: (1) Five-stage preprocessing pipeline that systematically transforms raw Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) data through exploratory analysis, data cleaning, vocabulary mapping, standardization, and ICU cohort extraction with comprehensive audit trails ensuring reproducibility. (2) Scalable parallel architecture that processes the entire 278.97 GB dataset in under 24 hours using 12 CPU cores and 64GB RAM through optimized chunked processing and parallel optimization, making large-scale multi-institutional data processing accessible to resource-constrained research teams. (3) Comprehensive benchmarks across four critical prediction tasks using multiple model architectures, establishing reproducible baselines for the research community. (4) Open-source implementation with complete code, documentation, and processed datasets, saving researchers months of preprocessing effort. 2 Related Work 2.1 Clinical Data Processing Pipelines As clinical datasets grow in scale and complexity, the need for standardized, reusable pipelines becomes increasingly critical. In particular, there has been many processing pipelines designed for MIMIC-III, MIMIC-IV, eICU and other EHR datasets. Notable contributions include MIMIC-Extract ( Wang et al. 2020 ) for cohort extraction, multitask benchmarks ( Harutyunyan et al. 2019 ) establishing standard prediction tasks, COP-E-CAT ( Mandyam et al. 2021 ) for modular preprocessing, an extensive MIMIC-IV pipeline ( Gupta et al. 2022 ) , METRE ( Liao and Voldman 2023 ) for cross-database validation, and reproducibility MIMIC benchmark ( Purushotham et al. 2018 ) . These pipelines have established a solid foundation for processing single-institution EHR datasets and have achieved remarkable success within their respective domains. However, extending them to multi-institutional environments poses several challenges. Existing pipelines are typically optimized for dataset-specific schemas (e.g., MIMIC’s custom structure or