Exploring Scaling Laws for EHR Foundation Models
Sheng Zhang, Qin Liu, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon · 2025
Read Exploring Scaling Laws for EHR Foundation Models by Sheng Zhang, Qin Liu, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22964.txt.
Introduction
Sheng Zhang and co-authors explore in 2025 whether the scaling rules seen for language models also hold for models trained on health records. Health records are rich, ordered, and common around the world, yet they differ in shape from plain text. The team trains transformer models on patient timelines from a large hospital data set, at varied sizes and costs.
Clear patterns appear as size and computing grow. Curves that hold cost fixed take a bent shape, and gains follow power laws that link cost, size, data, and clinical value. Readers learn that health record models scale in much the same way as language models, which can guide training choices.
Catalog record
| CREATOR | Sheng Zhang, Qin Liu, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Scaling Laws for EHR Foundation Models Sheng Zhang, Qin Liu, Naoto Usuyama, Cliff Wong, Tristan Naumann, Hoifung Poon Abstract The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) — a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare. 1 Introduction The discovery of scaling laws for large language models (LLMs) has significantly advanced the understanding of how model performance predictably improves as we scale up model size, dataset size, and computational resources for training ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . These scaling laws, initially characterized by training transformers on vast amounts of internet text data, have enabled the systematic development of increasingly powerful frontier models ( Brown et al. 2020 ; OpenAI 2024 ; OpenAI 2025 ) . Despite the explosion in data used to train LLMs, a substantial class of data remains largely untapped: electronic health records (EHRs). There are two primary barriers accounting for this underutilization. First, EHRs are proprietary and protected under strict privacy regulations, preventing open sharing and centralized access. Second, although EHRs contain text-based information, the longitudinal patient data comprising medical events and attributes are often standardized using specific coding systems, such as ICD codes ( Centers for Medicare & Medicaid Services 2025 ) for diagnoses and ATC codes ( World Health Organization 2025 ) for medications. This structured and codified information fundamentally differs from the natural language text typically processed by pretrained LLMs, presenting unique challenges for conventional LLMs. Nevertheless, EHRs share critical similarities with natural language text that make them amenable to transformer-based modeling ( Vaswani et al. 2023 ) . The events and attributes recorded in EHRs naturally form sequential timelines of patient encounters and treatments, analogous to word sequences in text data. Additionally, the volume at which EHRs are generated globally mirrors the scale of internet data, suggesting that foundation models trained on EHRs could benefit similarly from scaling laws. Motivated by these similarities and opportunities, our work provides an initial exploration into scaling laws specifically for EHR foundation models. We train transformer models of varying scales on patient timeline data derived from the MIMIC-IV database ( Johnson et al. 2023 ) , and systematically examine how model performance evolves with changes in model size, number of training tokens, and computational budgets. Our scaling-law experiments yield several key findings: • For a fixed computational budget (measured in floating-point operations, FLOPs), the IsoFLOPs curve — depicting the loss of EHR models relative to different model sizes — exhibits a clear parabolic relationship, consistent with scaling patterns observed in traditional LLMs. This curve enables estimation of the optimal model size that achieves minimal loss for a given computational budget. • By tracing the loss minima of IsoFLOP curves across varying compute budgets, we derive empirical power-law relationships linking compute, optimal model size, and optimal number of training tokens. These power laws enable us to predict performance improvements as model and data scales increase. Moreover, they reveal predictable scaling behavior and underscore the importance of maintaining a careful balance between model size and data volume. • Zero-shot evaluation on downstream healthcare applications shows strong correlation between lower validation loss and better predictive performance, reinforcing the validation loss of EHR modeling as a guiding metric for EHR model development. • Scaling model size lead to consistent improvements in clinical utility up to 28M parameters, beyond which performance begins to saturate due to insufficient training data. This plateau reflects the underfitting-overfitting tradeoff when models outscale the available token volume, and underscores the need for larger EHR datasets to sustain scaling gains. This preliminary characterization of scaling laws for EHR foundation models provides predictive insights into the expected performance improvements achievable by scaling model size, expanding EHR training datasets, and increasing computational resources. Ultimately, these scaling laws lay critical groundwork for realizing transformative advancements — akin to the “GPT moment” — in predicting key clinical outcomes such as mortality risk, ICU admissions, and hospital stays, thus paving the way for significant improvements in healthcare delivery and patient outcomes. 2 Methods 2.1 Problem Formulation Electronic health records (EHRs) consist of temporally ordered clinical events and attributes, forming a natural sequence akin to text in language modeling. Leveraging this sequential structure, we formulate the modeling of EHR data using an autoregressive framework similar to that employed in language models. Let a patient timeline from EHRs be denoted as a variable-length sequence of medical tokens {\mathbf{x}}=(x_{1},x_{2},\dots,x_{N}) , where each token x_{i} represents a distinct unit of clinical information, corresponding to diagnoses, medication administrations, hospital admissions, time intervals, or other meaningful elements from the patient’s health trajectory (see Section 2.3 ). Our goal is to model the joint probability distribution over the patient timeline p(x_{1},...,x_{N}) , which we factorize as a product of conditional probabilities ( Bengio et al. 2003 ) : p({\mathbf{x}})=\prod_{i=1}^{N}p(x_{i}\mid x_{1},...,x_{i-1}) To parameterize these conditional probabilities, we employ a decoder-only transfomer neural network ( Vaswani et al. 2023 ; Radford et al. 2018 ) with parameters \theta . The transformer model estimates each token’s likelihood given its preceding context as f_{\theta}(x_{1},...,x_{i})=p_{\theta}(x_{i}\mid x_{1},...,x_{i-1}) . The model is trained via a standard next-token prediction objective, which maximizes the following log-likelihood: \mathcal{L}({\mathbf{x}})=\sum_{i=1}^{N}\log p_{\theta}(x_{i}\mid x_{1},...,x_{i-1}) During inference, given a partial patient timeline (x_{1},...,x_{i}) , the model can autoregressively generate future tokens by sampling from the conditional distribution: (x_{i+1},...,x_{N})\sim p_{\theta}(x_{i+1},...,x_{N}\mid x_{1},...,x_{i})=\prod_{j=i+1}^{N}p_{\theta}(x_{j}|x_{1},...,x_{j-1}) This formulation enables flexible modeling of patient trajectories, and supports sequence completion for large-scale training and inference of EHR foundation models. 2.2 Database We choose MIMIC-IV ( Johnson et al. 2023 ) as the source database for our experiments. MIMIC is a large, publicly available database of EHRs that provides comprehensive clinical data for research purposes. The dataset encompasses de-identified health records of over 200,000 adult patients who were admitted to the emergency department or an intensive care unit (ICU) at Beth Israel Deaconess Medical Center in Boston, Massachusetts, between 2008 and 2019. MIMIC-IV is designed to reflect real-world hospital operations, with data collected and timestamped continuously across a wide range of clinical encounters. MIMIC-IV captures a rich array of data modalities, including: • Diagnoses recorded using ICD codes, • Medications prescribed and administered during hospital stays, • Laboratory test results and their associated values, • Procedures and clinical events , such as surgeries or imaging, • Vital signs and nursing assessments, • Demographics and admission information, • Outcomes , such as in-hospital and post-discharge mortality. All data entries are