Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs

Michael Wornow, Suhana Bedi, Miguel Angel Fuentes Hernandez, Ethan Steinberg, Jason Alan Fries, Christopher Ré, Sanmi Koyejo, Nigam H. Shah · 2024

Read Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs by Michael Wornow, Suhana Bedi, Miguel Angel Fuentes Hernandez, Ethan Steinberg, Jason Alan Fries, Christopher Ré, Sanmi Koyejo, Nigam H. Shah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16178.txt.

Introduction

Michael Wornow and seven co-authors test long context models on health records in a 2024 paper. A full patient record can run past ten thousand events. Most record models read under a thousand tokens at once. New architectures like Mamba promise far longer reading spans.

Their benchmark asks whether longer context truly helps prediction. The Mamba based model beat the prior best on nine of fourteen tasks. Three quirks of record data got special stress tests. Copied forward diagnoses create fake repetition inside records. Irregular gaps stretch a window across wildly different time spans. Later record parts grow harder as diseases accumulate. The study maps where long context pays off and where it strains. Clinical use needs that robustness, not raw scores alone.

Catalog record

CREATORMichael Wornow, Suhana Bedi, Miguel Angel Fuentes Hernandez, Ethan Steinberg, Jason Alan Fries, Christopher Ré, Sanmi Koyejo, Nigam H. Shah
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs Michael Wornow, Suhana Bedi, Miguel Angel Fuentes Hernandez, Ethan Steinberg, Jason Alan Fries, Christopher Ré, Sanmi Koyejo, Nigam H. Shah Abstract Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, most existing EHR FMs have context windows of < < 1k tokens. This prevents them from modeling full patient EHRs which can exceed 10k’s of events. Recent advancements in subquadratic long-context architectures (e.g., Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance – our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. For clinical applications, however, model performance alone is insufficient – robustness to the unique properties of EHR is crucial. Thus, we also evaluate models across three previously underexplored properties of EHR data: (1) the prevalence of “copy-forwarded” diagnoses which creates artificial repetition of tokens within EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance (e.g., a 14% higher Brier loss when making predictions for the most versus least irregular patients), but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study for identifying new challenges in modeling sequential data motivated by domains outside of natural language. We release our model checkpoints and code at: https://github.com/som-shahlab/long_context_clues 1 Introduction Figure 1: The central claims of this paper. (a) EHRs are sequences: An EHR is simply a timeline of clinical events that occur to a patient, and thus can be naturally represented as a sequence of tokens. (b) Long context improves performance: AUROC on clinical prediction tasks tends to increase with longer context lengths, with Hyena ( red ) being the notable exception. Overall, Mamba ( green ) at a context length of 16k achieves the highest average AUROC across 14 diverse clinical prediction tasks. (c) EHR data has distinct properties: In contrast to natural language, EHR data has unique properties whose implications remain under-explored in the ML literature. Here, we highlight three such attributes – copy-forwarding, irregular time intervals between tokens, and disease progression. (d) EHRs properties present unique modeling challenges: Stratifying patients by the degree to which they exhibit each EHR-specific property, we find that higher Brier scores (i.e., worse model performance) are associated with patients who have more repetitive ( top ) or irregular ( middle ) EHRs. Additionally, the perplexity of tokens later in a patient’s timeline tends to be higher, even when conditioning on prior tokens (bottom). Foundation Models (FMs) ( Bommasani et al., 2021 ) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks ( Odgaard et al., 2024 ; Yang et al., 2023 ) . Such models can improve patient outcomes via early detection of disease and risk stratification ( Steinberg et al., 2023 ) . As an EHR is simply a list of chronologically-ordered clinical events (see Figure 1 a), it can be modeled as a sequence of tokens. Instead of subwords or image patches, however, tokens represent clinical events like diagnoses and procedures ( McDermott et al., 2023 ) . This approach has enabled the application of transformer architectures originally developed for natural language processing (NLP) such as BERT ( Rasmy et al., 2021 ; Li et al., 2020 ; Odgaard et al., 2024 ) and GPT ( Steinberg et al., 2021 ; Pang et al., 2024 ; Kraljevic et al., 2024 ) to EHR data. A critical choice in FM design is context length – i.e. how many tokens of input the model can ingest. Longer context lengths have shown a consistent positive impact on FM performance across various domains by enabling models to reference and reason over more information ( Xiong et al., 2023 ) . Given the typical hospital’s limited compute resources, however, transformer-based EHR FMs have been limited to processing short context lengths (i.e., 512 tokens) due to the quadratic scaling of attention with input length ( Vaswani et al., 2017 ) . As a single patient’s EHR can contain 10k’s of tokens, this greatly limits the amount of data that EHR FMs can consider. This is especially true for the sickest patients – i.e. the ones of most interest to a hospital for prediction tasks – as they typically have high healthcare utilization and thus have very long timelines, as can be seen in the CDF plots of patient sequence length in Appendix Figure 6 . Recently developed subquadratic architectures such as Mamba ( Gu & Dao, 2024 ) and Hyena ( Poli et al., 2023a ) that are optimized for long contexts offer a potential solution. As EHR FMs begin driving real-world care decisions, it is essential to better understand the implications of adapting these long context architectures for clinical prediction making. However, their effectiveness on EHR data remains unclear . In contrast to natural language, EHR data exhibits specific types of token repetition and noise that complicate the expected benefits of longer contexts. We identify and present the first quantitative analysis of three such underexplored properties, as outlined in Figure 1 c: 1. Copy-forwarding — key diagnoses are repeated across multiple visits due to billing practices, leading to artificial repetition of tokens in the EHR ( Thornton et al., 2013 ) . 2. Irregular time intervals between tokens — unlike in natural language where consecutive tokens are trivially 1 position apart, consecutive clinical events can be days or years apart, thus creating a wide range of timescales within a single context ( McDermott et al., 2023 ) . 3. Disease progression — later tokens in a patient’s timeline are harder to predict as disease complexity tends to increase with age ( Fabbri et al., 2015 ) , even when conditioning on prior tokens; this contrasts with natural language, in which later tokens in a prompt tend to exhibit lower perplexities ( Peng et al., 2023b ) . While several papers have introduced transformer-based EHR FMs, they typically only evaluate at a single context length of 512 tokens, as shown in Table 1 . Evaluations of subquadratic architectures on EHR data have also been limited to one context length and do not consider “longitudinal” (i.e. full-length) EHRs ( Fallahpour et al., 2024 ) . To our knowledge, there has been no systematic evaluation of the impact of context length on state-of-the-art transformer and non-transformer architectures trained on longitudinal EHR data for clinical prediction tasks. To address these gaps in the literature, our paper makes the following three contributions: • State-of-the-art (SOTA) Clinical Prediction Making with Subquadratic Architectures: We train and evaluate

Read the complete text