FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records

Chao PangVincent JeanselmeYoung Sang ChoiXinzhuo JiangZilin JingAparajita KashyapYuta KobayashiYanwei LiFlorent PolletKarthik NatarajanShalmali JoshiDepartment of Biomedical InformaticsColumbia University · 2025

Read FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records by Chao PangVincent JeanselmeYoung Sang ChoiXinzhuo JiangZilin JingAparajita KashyapYuta KobayashiYanwei LiFlorent PolletKarthik NatarajanShalmali JoshiDepartment of Biomedical InformaticsColumbia University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16941.txt.

Introduction

Jeanselme and colleagues test large pre trained models for health records in a benchmark called FoMoH in twenty twenty-five. Such models promise to work with little labeled data and to move well between hospitals. No shared test had checked if they keep those promises. The team built fourteen tasks that matter in clinics, from outlook to early warning of illness.

They tested six top models on records of more than six million patients from two sources. The best models beat older methods when labeled data were scarce. They also scored more fairly across patient groups. But they did worse on rare conditions and moved poorly to a new hospital. Score trust also fell when labels were scarce.

Catalog record

CREATORChao PangVincent JeanselmeYoung Sang ChoiXinzhuo JiangZilin JingAparajita KashyapYuta KobayashiYanwei LiFlorent PolletKarthik NatarajanShalmali JoshiDepartment of Biomedical InformaticsColumbia University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records Chao PangVincent JeanselmeYoung Sang ChoiXinzhuo JiangZilin JingAparajita KashyapYuta KobayashiYanwei LiFlorent PolletKarthik NatarajanShalmali JoshiDepartment of Biomedical InformaticsColumbia University Abstract Foundation models hold significant promise in healthcare, given their capacity to extract meaningful representations independent of downstream tasks. This property has enabled state-of-the-art performance across several clinical applications trained on structured electronic health record (EHR) data, even in settings with limited labeled data, a prevalent challenge in healthcare. However, there is little consensus on these models’ potential for clinical utility due to the lack of desiderata of comprehensive and meaningful tasks and sufficiently diverse evaluations to characterize the benefit over conventional supervised learning. To address this gap, we propose a suite of clinically meaningful tasks spanning patient outcomes, early prediction of acute and chronic conditions, including desiderata for robust evaluations. We evaluate state-of-the-art foundation models on EHR data consisting of 5 5 million patients from Columbia University Irving Medical Center (CUMC), a large urban academic medical center in New York City, across 14 14 clinically relevant tasks. We measure overall accuracy, calibration, and subpopulation performance to surface tradeoffs based on the choice of pre-training, tokenization, and data representation strategies. Our study aims to advance the empirical evaluation of structured EHR foundation models and guide the development of future healthcare foundation models. 1 Introduction Foundation Models [ 8 ] have demonstrated state-of-the-art performance across natural language [ 76 ] , multimodal vision-language [ 15 ] , multivariate time-series [ 16 ] , and biological data [ 20 ] . Pre-trained on large amounts of unlabelled data, these models learn meaningful representations from simple tasks such as next-token prediction or masked language modeling [ 17 , 26 , 59 ] . These learned representations have proven suitable for diverse downstream tasks, in contrast with the conventional supervised learning paradigm in which models are trained for specific downstream tasks [ 59 , 74 , 23 , 52 ] . Increasing availability of digitized structured Electronic Health Records (EHRs), reflective of patients’ interactions with the healthcare system, offers an opportunity to leverage foundation models in healthcare [ 84 ] , where labeled data are often sparse. Multiple foundation models have tackled the unique challenges associated with EHR, such as temporality [ 56 , 67 ] and long context windows [ 85 ] , presenting competitive performance across medical tasks [ 8 , 24 ] . Figure 1: FoMoH is an evaluation of state-of-the-art structured EHR foundation models using (i) clinically meaningful and diverse downstream tasks, (ii) patient cohorts derived to prevent data leakage, (iii) augmented evaluation beyond overall discrimination, such as calibration, and fairness along clinically relevant factors such as healthcare utilization. Despite the proliferation of EHR foundation models, there is a critical lack of consensus on optimal design choices and their potential for clinical utility. Clinically-grounded evaluation of structured EHR foundation models is essential to quantify the impact of specific design choices and a standardized measure of progress [ 8 ] . Current evaluations have been large-scale but restricted to limited metrics, downstream tasks such as patient mortality, and limited by the lack of data-sharing and interoperability across health institutions [ 62 , 84 ] . While data sharing is still challenging, several efforts to harmonize EHR data and standardize AI pipelines have removed some barriers to foundation model evaluations in EHRs [ 3 , 28 , 39 , 63 , 87 ] . Nonetheless, existing benchmarks have focused on performance evaluation along limited design choices such as context-length or model sizes, and evaluating population-level metrics [ 84 ] . Evaluating the downstream impact of these choices requires a comprehensive evaluation of population and subpopulation performance, calibration, and discrimination on standardized and diverse clinically meaningful downstream prediction tasks. We propose a clinically grounded evaluation of structured EHR foundation models on EHR data from Columbia University Irving Medical Center (CUMC), a large academic medical center that caters to a diverse patient population. To ensure reproducibility and transportability, we build on the increasingly robust infrastructure for standardizing EHR data, downstream tasks, and foundation model evaluations [ 3 , 39 , 87 ] 1 1 1 https://www.ohdsi.org/data-standardization/ . Our work’s primary contributions, illustrated in Figure 1 , are: 1. Clinically-grounded prediction task desiderata. The downstream utility of EHR foundation models will be determined by reliable prediction for patient prognosis for healthcare operations (e.g., readmission), patient outcomes, early diagnosis of diverse conditions and pathologies, e.g., chronic conditions, and acute events. We introduce 14 curated and clinically-meaningful downstream prediction tasks consisting of 11 phenotypes and 3 patient prognoses, with ACES -compatible representations [ 87 ] — a library for automatic extraction — for reproducibility and transportability across healthcare institutions. 2. Evaluating foundation models on structured EHR. We create a MEDS-compatible dataset --- a standardized EHR format --- to evaluate major foundation models spanning key design choice differences, such as tokenization and pre-training strategies developed at major healthcare institutions, including the assessment of transportability of EHR-based foundation models. The proposed evaluation enables a robust, fair, and clinically meaningful comparison across these design choices, with manual and automated curation to prevent data contamination between pre-training, linear probing, and downstream evaluation. Our benchmark, available on Github 2 2 2 https://github.com/reAIM-Lab/ehr_foundation_model_benchmark/ , is intended as a prescriptive framework for future assessments of structured and unstructured EHR foundation models. 3. Desiderata of evaluation metrics of EHR foundation models. Motivated by downstream clinical utility, we present a broad set of metrics to capture clinically relevant measures of performance, going beyond discriminative performance, often used as the only evaluation measure in foundation model literature. We measure calibration and fairness of aspects unique to healthcare, such as healthcare utilization. As deployment is a critical bottleneck in healthcare, we evaluate the computational pre-training cost and the transportability of EHR foundation models without local fine-tuning. 2 Related work There has been a proliferation of unimodal and multimodal foundation models in health and medicine, leveraging structured and unstructured EHR, imaging, genomic, -omics, and other translational medical data, building on the recent success of natural language and vision-based foundation models trained on internet-scale data [ 52 , 46 , 23 , 29 , 86 , 85 , 84 , 18 , 20 , 75 ] . In this work, we focus on benchmarking EHR-based foundation models, leveraging their structured components (see Table 3 in Appendix A , which describes the datasets, tasks and evaluations of existing EHR foundation models). Structured EHR Foundation Models. A patient’s electronic health record contains clinical events in chronological order, including tokenized occurrences of labs, procedures, prescriptions, and visit types. Consequently, a patient trajectory is a sequence of irregularly sampled events, analogous to natural language. This parallel has led to adapting the language modeling paradigm to EHR. However, EHR-specific challenges require additional innovations in tokenization, data representation, model architectures, and choice of pre-training tasks, for both unstructured and structured data. First, EHRs are marked by irregular sampling, partially informative of the patient’s physiology, confounded by the context of patient healthcare utilization patterns, and healthcare processes [ 1 , 6 , 5 , 11 , 32 ] . Second, EHR tokens may be associated with numerical values from lab tests and other measurements. Third, EHRs cover multiple

Read the complete text