ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling

Yuanyun Zhang, Shi Li · 2025

Read ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling by Yuanyun Zhang, Shi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07373.txt.

Introduction

Yuanyun Zhang and Shi Li present ChronoFormer, a transformer design built to model clinical events with their timing, in a 2025 paper. Health records unfold over time, and that time structure makes outcome prediction hard for standard models. Their design encodes time directly, using time embeddings, layered attention, and masking shaped by medical knowledge.

The team tested it on three benchmark tasks: predicting death, predicting readmission, and forecasting the onset of long-term conditions. ChronoFormer beat the current best methods by clear margins, they report. Reading its attention patterns showed it catching long-range links between events that carry clinical meaning, a sign the time handling works as intended.

Catalog record

CREATORYuanyun Zhang, Shi Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07373

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling Yuanyun Zhang, Shi Li Abstract The temporal complexity of electronic health record (EHR) data presents significant challenges for predicting clinical outcomes using machine learning. This paper proposes ChronoFormer , an innovative transformer-based architecture specifically designed to encode and leverage temporal dependencies in longitudinal patient data. ChronoFormer integrates temporal embeddings, hierarchical attention mechanisms, and domain-specific masking techniques. Extensive experiments conducted on three benchmark tasks—mortality prediction, readmission prediction, and long-term comorbidity onset—demonstrate substantial improvements over current state-of-the-art methods. Furthermore, detailed analyses of attention patterns underscore ChronoFormer’s capability to capture clinically meaningful long-range temporal relationships. 1 Introduction The digitization of healthcare data provides immense potential for machine learning in clinical applications, yet EHR data’s irregularity, sparsity, and complexity make it challenging to effectively model longitudinal patient trajectories ( Seymour et al. 2012 ) . Existing approaches such as GRU-based and attention-based models have partially addressed these challenges but fail to fully capture temporal nuances, particularly in scenarios with heterogeneous time intervals and variable-length histories ( Shickel et al. 2017 ; Steinberg et al. 2021 ; Yang et al. 2022 ) . Recent studies convert structured EHR data into textual formats ( Hegselmann et al. 2023 ; Lee et al. 2024 ; Hegselmann et al. 2025 ; Gao et al. 2024 ; Ono and Lee 2024 ) , facilitating their use with transformer-based language models. This reformulation allows models pretrained on natural language to be adapted to clinical sequences, leveraging contextual embedding power. While promising, these methods typically overlook detailed temporal modeling, often reducing timestamps to coarse positional encodings or relying on event order alone, which discards essential information about inter-event intervals and rate of change—both critical in clinical reasoning. Addressing this gap, our work introduces ChronoFormer , which explicitly incorporates time-awareness into the transformer architecture via continuous-time encoding mechanisms that preserve temporal distances across visits and events. ChronoFormer leverages a hybrid token-time representation where both clinical concepts and their timestamps are encoded jointly, allowing for fine-grained control over the attention mechanism’s temporal bias. This enables the model to differentiate between, for instance, frequent but clinically insignificant fluctuations and slow, meaningful drifts in patient status. Empirically, we demonstrate that ChronoFormer achieves superior performance across a suite of clinical prediction tasks—ranging from mortality risk estimation to medication forecasting—outperforming both time-agnostic transformers and temporal RNN baselines ( Medsker et al. 2001 ; Dey and Salem 2017 ) . By tightly integrating temporal structure into the foundation of the model, we move toward more faithful representations of patient timelines, bridging the gap between statistical learning and clinical temporal reasoning. 2 Related Work Language models adapted for biomedical and clinical domains have significantly advanced the state of natural language processing in healthcare. Early efforts such as BioBERT ( Lee et al. 2020 ) and ClinicalBERT ( Alsentzer et al. 2019 ) demonstrated that domain-specific pretraining on biomedical literature and clinical notes improves performance on tasks such as named entity recognition, relation extraction, and clinical text classification. More recent architectural innovations, exemplified by Clinical ModernBERT ( Lee et al. 2025 ) , integrate efficient attention mechanisms, rotary positional encodings, and deeper contextual representations, allowing the model to scale to longer inputs and more complex clinical narratives ( Zhang et al. 2020 ) . These models are typically trained on unstructured data, focusing on syntactic and semantic aspects of language, but often fall short when applied directly to structured, temporal data common in EHR systems. Other methods have also been explored in this space including ( Chen et al. 2024 ; Kraljevic et al. 2024 ; Darabi et al. 2020 ) To bridge the gap between unstructured and structured data modalities, recent research has explored EHR records as event-based sequences that can be ingested by language models. This approach allows pretrained transformers to be fine-tuned on structured clinical data with minimal architectural modification, enabling transfer learning and zero-shot inference on downstream tasks. Frameworks such as BEHRT ( Li et al. 2020 ) , Med-BERT ( Rasmy et al. 2021 ) , and CEHR-BERT ( Pang et al. 2021 ) exemplify this trend, encoding visits as token sequences and leveraging transformer encoders for predictive modeling. However, these methods often rely on fixed-length positional encodings or coarse-grained time binning, which limits their ability to capture fine-grained temporal irregularity—a key property of real-world EHRs. The introduction of the Medical Event Data Standard (MEDS) by Arnrich et al. 2024 represents a crucial step toward standardizing clinical event representations across institutions and use cases. MEDS proposes a consistent schema for encoding structured EHR data into temporally-aware event sequences, facilitating interoperability and large-scale modeling. It enables researchers to abstract away from the idiosyncrasies of individual EHR systems and focus on high-level temporal dynamics. However, most transformer-based models using MEDS-formatted data still treat time as an auxiliary input or incorporate it indirectly, rather than making it a first-class component of the modeling architecture. Our work builds on these foundations by explicitly integrating temporal modeling into the core of the transformer architecture. In contrast to prior approaches that use absolute or relative positional encodings as a proxy for time, ChronoFormer introduces a continuous-time representation that modulates self-attention weights based on inter-event intervals. This design allows the model to adaptively focus on temporally relevant events and maintain sensitivity to clinical pacing. By leveraging the representational power of transformer models and augmenting them with principled time-awareness, we significantly advance the capabilities of EHR-based predictive modeling. 3 Methodology 3.1 Data Standardization To facilitate robust modeling of temporal patient trajectories, we standardize EHR sequences using the Medical Event Data Standard (MEDS) framework ( Arnrich et al. 2024 ) . Each patient’s clinical history is transformed into a sequence of temporally ordered bins, where each bin B_{i} represents a fixed-length time window (e.g., 24 hours) and contains a set of atomic clinical events \{e_{j}^{(i)}\} that occurred within the window. Events span multiple modalities including diagnosis codes (ICD), procedure codes (CPT), medication administrations (RxNorm), and laboratory results (LOINC) as these concepts are difficult to learn ( Lee and Lindsey 2024 ) . For each event e_{j}^{(i)} , we associate a fine-grained timestamp t_{j}^{(i)} , reflecting the actual time of occurrence, and optional metadata m_{j}^{(i)} such as dosage, lab values, or procedural modifiers. This binning process allows us to retain high temporal resolution while enabling transformer-based modeling at the bin level, thus maintaining scalability. Events are encoded as discrete token IDs based on medical ontologies, and the resulting structured sequences (e_{j}^{(i)},t_{j}^{(i)},m_{j}^{(i)}) form the input to our model. 3.2 ChronoFormer Architecture ChronoFormer is a transformer-based architecture designed to capture both local (intra-visit) and global (inter-visit) temporal dependencies in longitudinal structured health records. It introduces two primary innovations over conventional transformer designs: temporally aware embeddings and hierarchical attention. These modifications enable ChronoFormer to model the asynchronous, sparse, and multiscale nature of clinical data with greater fidelity. Temporal Embeddings: Unlike conventional transformers that rely on discrete positional embeddings to encode order, ChronoFormer explicitly incorporates both absolute and relative temporal information through a dual temporal embedding mechanism ( Nguyen et al. 2018 ) . Each event e_{j}^{(i)} is augmented with a continuous-time embedding that reflects its temporal positioning in two ways: the absolute timestamp t_{j}^{(i)} and the relative time delta

Read the complete text