Tokenization Tradeoffs in Structured EHR Foundation Models
Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung · 2026
Read Tokenization Tradeoffs in Structured EHR Foundation Models by Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15644.txt.
Introduction
Guo and co-authors test how token design shapes health record foundation models. These models learn from long timelines of timed clinical events. The team pretrained a transformer on pediatric records under a factorial design. They varied event coding, time coding, and workflow notes. Joint event coding and positional time coding won on nearly all 74 prediction tasks. Both also cut pretraining compute, by 39.5 and 9.6 percent. A test on adult ICU data showed the joint coding gain carries over.
Catalog record
| CREATOR | Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73234 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tokenization Tradeoffs in Structured EHR Foundation Models Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung Abstract Foundation models for structured electronic health records (EHRs) are pretrained on longitudinal sequences of timestamped clinical events to learn adaptable patient representations. Tokenization – how these timelines are converted into discrete model inputs – determines what information is preserved, how efficiently it is encoded, and which relationships must be learned versus pre-computed. Yet the impact of tokenization design choices on downstream performance and computational efficiency remains largely unexplored. Here, we pretrained a transformer on pediatric EHR data under a factorial design, varying tokenization along event encoding, time encoding, and workflow annotation. We evaluated area-under-the-receiver-operating-characteristic curve across 74 clinical prediction tasks. Joint event encoding and positional time encoding outperformed their alternatives (73/74 and 71/74 tasks) while requiring 39.5% and 9.6% fewer pretraining floating-point operations, respectively. Targeted ablations traced the joint encoding advantage to local binding efficiency, that is, code-attribute pairs are combined into single tokens, rather than split across tokens that the model must learn to associate during pretraining. External evaluation on an adult intensive care unit cohort demonstrated that this advantage generalizes despite substantial vocabulary mismatch, while temporal and workflow effects remain institution-specific. These results establish tokenization as a tractable lever for improving both the performance and efficiency of EHR foundation models. 1 Introduction Foundation models are transforming artificial intelligence (AI) in healthcare by shifting development from bespoke, single-purpose models to reusable, adaptable backbones. [ 3 , 22 , 44 , 7 , 33 ] Foundation models for structured electronic health records (EHR), trained on longitudinal sequences of timestamped clinical events such as diagnoses, procedures, medications, and laboratory results, have demonstrated improved prediction performance, [ 35 , 34 ] robustness to distribution shifts, [ 14 , 13 , 20 ] and sample efficiency [ 42 ] in downstream clinical tasks. As these models advance toward broader clinical adoption, [ 43 ] understanding the design decisions that shape their learned representations becomes critical for guiding principled model development. An important yet largely understudied step in training EHR foundation models is tokenization: converting a patient’s clinical timeline into a sequence of discrete tokens. Because clinical events combine categorical codes, continuous measurements, and temporal relationships, the tokenization strategy determines what information is preserved in the token sequence (e.g., code-attribute associations, temporal intervals), how much of the patient’s history fits within a fixed context window or compute budget, and which relationships the model must discover from data versus pre-computed during tokenization. Tokenization choices are fixed at pretraining time and propagate to every downstream application, making even modest effects consequential. Three tokenization design axes recur across existing structured EHR foundation models. First, event encoding determines how individual clinical events are represented as tokens. An elevated serum glucose measurement, for example, may be encoded as a single composite token that fuses the code with its value (joint encoding), [ 35 , 18 ] or decomposed into separate tokens that the model processes sequentially or aggregates prior to input (factorized encoding). [ 31 , 40 , 15 , 1 ] Second, time encoding determines how temporal information is conveyed: whether through dedicated time interval tokens consuming sequence positions, [ 31 , 26 , 25 ] positional encodings calibrated to temporal intervals, [ 35 ] or implicitly through sequence order alone. [ 41 , 19 , 29 ] Third, the scope of clinical context determines whether to represent clinical actions as single events (e.g., a finalized laboratory result) or as multi-step workflow sequences capturing process stages such as order placement, specimen collection, and result finalization. These choices jointly determine the vocabulary size, sequence length, and pretraining compute requirements of a model, and may further affect downstream task performance and cross-institutional generalizability. Yet these axes have been adopted without controlled comparison, hindering robust best practices. [ 11 ] Encoding clinical structure at tokenization time versus requiring the model to learn it from data reflects a fundamental trade-off in data-limited settings. EHR foundation models are pretrained on orders of magnitude fewer tokens than frontier language models, [ 43 , 11 ] limiting the signal available to learn complex associations. Binding at tokenization time hard-codes compositional structure but expands the vocabulary and increases parameters in the embedding layer. Factorized encoding preserves compositionality and a compact vocabulary but requires the model to learn code-value associations from pretraining data. For instance, joint event encoding pre-computes the association between a clinical code and its measured value within a single token, whereas factorized encoding requires the model to learn that the same value token carries different clinical meaning depending on which code it follows. This is a form of the binding problem in neural networks, [ 10 ] applied locally to adjacent tokens that must be interpreted together, which we refer to as local binding efficiency. Whether this trade-off yields measurable performance differences has not been well established for EHR foundation models. Here, we quantify how tokenization design choices shape EHR foundation model performance and efficiency. Using a 2\times 2\times 2 factorial experiment, we estimate the independent effects of event encoding, time encoding, and workflow annotations across 74 clinical prediction tasks spanning six clinical domains. We complemented this analysis with targeted ablation experiments to probe mechanisms and external evaluations to assess cross-institutional generalizability. Models with a fixed transformer architecture were pretrained on pediatric EHR data and evaluated locally and on an external adult intensive care cohort. [ 16 ] Our key findings are as follows: 1. Joint event encoding and positional time encoding improved downstream discrimination while requiring less pretraining compute, indicating that encoding more clinical structure at tokenization time improves both performance and efficiency. 2. Ablations traced the joint encoding advantage to local binding efficiency, whereby code-attribute pairs are pre-computed as single tokens during tokenization rather than learned across tokens from limited pretraining data. 3. Positional time encodings calibrated to temporal intervals showed modest improvement over sequence order alone, while dedicated time tokens degraded performance, indicating that how time is encoded matters more than whether it is encoded. 4. Benefits tied to institution-specific properties showed limited transfer: workflow annotations reflect local practice, and positional time encodings calibrated to one patient population did not generalize to populations with different age distributions. 2 Methods 2.1 Hospital Datasets This study used EHR data from The Hospital for Sick Children (SickKids), a tertiary pediatric hospital, as the primary development site, with data from Beth Israel Deaconess Medical Center (BIDMC), an adult academic medical center with intensive care unit admissions, used for external evaluation (MIMIC-IV). The SickKids dataset was sourced from the SickKids Enterprise-wide Data in Azure Repository (SEDAR), [ 12 ] which consolidates EHR data from SickKids’ Epic Clarity database into a standardized schema of 20 clinically organized tables. EHR data were mapped to the Medical Event Data Standard (MEDS) [ 2 , 36 ] format with clinical concepts standardized to Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) ontologies. The MIMIC-IV dataset (version 1.0), [ 16 ] contains de-identified EHR data from patients admitted to the intensive care unit or emergency department at BIDMC between 2008 and 2019. MIMIC data were mapped to the OMOP CDM using code from the