Temporal Entailment Pretraining for Clinical Language Models over EHR Data

Tatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li · 2025

Read Temporal Entailment Pretraining for Clinical Language Models over EHR Data by Tatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18128.txt.

Introduction

Tatsunori Tanaka and co-authors teach clinical language models to reason over time in patient records in two thousand twenty-five. Models trained on hospital notes treat each record as a still page. Real care unfolds in order, with later states growing out of earlier ones.

Their pretraining task pairs an earlier note segment with a later one and asks: does the later state follow from, clash with, or sit neutral to the earlier? Learning that judgment builds quiet time sense into the model. Trained on a large corpus drawn from the MIMIC-IV records, the model topped tests in time-aware question answering, early warning prediction, and disease course modeling. The task is offered as a general add-on for clinical models.

Catalog record

CREATORTatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temporal Entailment Pretraining for Clinical Language Models over EHR Data Tatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li Abstract Clinical language models have achieved strong performance on downstream tasks by pretraining on domain-specific corpora such as discharge summaries and medical notes. However, most approaches treat the electronic health record (EHR) as a static document, neglecting the temporally-evolving and causally-entwined nature of patient trajectories. In this paper, we introduce a novel temporal entailment pretraining objective for language models in the clinical domain. Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state. Through this temporally-structured pretraining task, models learn to perform latent clinical reasoning over time, improving their ability to generalize across forecasting and diagnosis tasks. We pretrain on a large corpus derived from MIMIC-IV and demonstrate state-of-the-art results on temporal clinical QA, early warning prediction, and disease progression modeling. 1 Introduction Electronic health records (EHRs) ( Seymour et al. 2012 ; Kim et al. 2019 ) are among the most temporally and semantically dense data sources available for machine learning ( Rajpurkar et al. 2022 ) . A single patient’s trajectory may span years of clinical encounters, capturing symptoms, diagnostics, interventions, and outcomes, each timestamped and variably documented in structured and unstructured formats ( Zhao et al. 2019 ; Huguet et al. 2020 ) . This makes EHRs a fertile ground for developing temporally-aware machine learning models that can reason about patient states, anticipate outcomes, and provide assistive decision support. Yet, despite this richness, current language models trained on EHR data often fail to incorporate the intrinsic temporality of the clinical domain in a meaningful way. The prevailing paradigm in clinical NLP has relied heavily on masked language modeling (MLM) objectives ( Salazar et al. 2019 ) , adapted from general-purpose models like BERT ( Devlin et al. 2019 ) . While effective for learning surface-level co-occurrence statistics, MLM does not naturally encode directional or causal relationships across time, nor does it encourage the model to understand how earlier clinical states influence later outcomes. As a result, downstream tasks that require longitudinal reasoning—such as forecasting disease progression, generating temporal summaries, or performing early warning detection—often require task-specific architectures or external supervision. In contrast to these approaches, we argue that clinical language models should be explicitly trained to reason over time, not merely across context windows. Just as human clinicians form diagnostic impressions based on how symptoms evolve or resolve over time, so too should models be equipped to interpret EHRs as temporally ordered sequences where earlier events imply, contradict, or are agnostic to subsequent ones. This motivates a shift away from static snapshot modeling and toward representations that encode temporal entailment. Drawing inspiration from natural language inference (NLI)—a central task in general-domain NLP that captures semantic relationships between text pairs—we introduce a new self-supervised objective tailored to the clinical domain: Temporal Entailment Pretraining (TEP) . Rather than predicting masked tokens or next notes, we construct sentence pairs from temporally-separated segments of the same patient’s record. The model must determine whether the later segment is logically entailed by, contradicts, or is neutral with respect to the earlier segment. Unlike standard NLI, which depends on curated corpora such as SNLI or MNLI ( bowman2015large ; williams2018broad ) , our supervision is derived automatically from clinical temporal patterns, leveraging weak heuristics, diagnostic staging progressions, and synthetic augmentation. This formulation yields several advantages. First, it imposes a directional, asymmetric relationship between input pairs, mirroring the causal and diagnostic dependencies clinicians often infer. Second, it induces temporally-aligned representation spaces that naturally capture clinical trajectories ( Steinberg et al. 2023 ) . Third, it generalizes across data modalities: the EHR segments can encode free-text notes ( Huang et al. 2019a ; Gabriel et al. 2018 ) , structured codes ( Lee and Lindsey 2024 ) , lab-derived summaries ( Hegselmann et al. 2023 ; Lee et al. 2024b ; Ono and Lee 2024 ) , or even mixed representations (multimodal) ( Yang and Wu 2021 ) . Finally, it scales, enabling pretraining over millions of temporal pairs without manual annotation. Through TEP, we bridge the gap between temporal reasoning and language model pretraining in the clinical domain. We demonstrate that models trained with this objective not only improve performance on traditional clinical NLP benchmarks but also exhibit stronger generalization and temporal calibration on tasks requiring sequential prediction. Ultimately, our work suggests a new direction for language model development in healthcare: one where temporality is a core part of the pretraining signal, not an afterthought. 2 Related Work 2.1 Clinical Language Models The development of domain-specific language models has significantly advanced clinical natural language processing (NLP). Early efforts, such as ClinicalBERT ( Alsentzer et al. 2019 ) , adapted BERT to the clinical domain by fine-tuning on MIMIC-III notes ( Johnson et al. 2016 ) , resulting in improved performance on tasks like named entity recognition and medical natural language inference (MedNLI). Building upon this, Clinical ModernBERT ( Lee et al. 2025 ) introduced architectural enhancements from modernbert ( Warner et al. 2024 ) , including rotary positional embeddings and Flash Attention, and extended context lengths up to 8,192 tokens. Pretrained on a diverse corpus comprising PubMed abstracts, MIMIC-IV clinical data, and medical ontologies, Clinical ModernBERT demonstrated superior performance on long-context biomedical NLP tasks, reaffirming the efficacy of encoder-only architectures in clinical settings. There are other variants with other modifications, some of which we will highlight here: MedBERT ( Liu et al. 2021 ) , ClinicalBERT ( Huang et al. 2019b ) , Clincial Longformer ( Li et al. 2022 ) . 2.2 Natural Language Inference in Clinical Contexts Natural Language Inference (NLI) has emerged as a critical task in clinical NLP, enabling models to reason about entailment relationships between clinical statements ( Herlihy and Rudinger 2021 ) . The MedNLI dataset ( Romanov and Shivade 2018 ) was specifically designed to evaluate NLI in the clinical domain, highlighting the challenges posed by domain-specific language and the necessity for specialized models. Subsequent studies have leveraged transformer-based architectures, such as ClinicalBERT ( Huang et al. 2019b ) , to improve NLI performance on MedNLI, underscoring the importance of domain adaptation in clinical reasoning tasks. 2.3 Temporal Modeling in Clinical NLP Temporal dynamics are intrinsic to clinical narratives, yet many models inadequately capture this aspect. Recent approaches have sought to integrate temporal reasoning into clinical NLP models. For instance, methods that model disease progression or predict future clinical events necessitate an understanding of temporal relationships within patient records. We will acknowledge that aside from Clinical NLP methods, there has been a lot of work in the EHR foundation model space ( Wornow et al. 2023b ) that has effectively integrated temporal aspects into the modeling framework ( McDermott et al. 2023 ; Steinberg et al. 2023 ; Pang et al. 2021 ) . However in the natural language format, this is much more difficult to represent. Our proposed Temporal Entailment Pretraining (TEP) framework builds upon these efforts by explicitly modeling temporal entailment between patient states, thereby enhancing the model’s capacity for longitudinal

Read the complete text