Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification

Rachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, Ole Winther · 2025

Read Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification by Rachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, Ole Winther (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12418.txt.

Introduction

Rachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, and Ole Winther wrote this paper in two thousand twenty-five. It classifies patients from electronic health records with a transformer that reads along two axes at once.

Health records keep growing: more fields, longer spans, more data kinds. Transformers fit such bulk, but the way records are fed in can waste the gaps, which often carry signal. Their Bi-Axial Transformer attends to both the clinical fields and the time points. On sepsis warning it sets the best score known. On death risk it matches the top methods. It shrugs off missing data better than rival transformers. Its learned sensor codes can also move to new tasks.

Catalog record

CREATORRachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, Ole Winther
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12418

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification Rachael DeVries, Casper Christensen, Marie Lisandra Zepeda Mendoza, Ole Winther Abstract Electronic Health Records (EHRs), the digital representation of a patient’s medical history, are a valuable resource for epidemiological and clinical research. They are also becoming increasingly complex, with recent trends indicating larger datasets, longer time series, and multi-modal integrations. Transformers, which have rapidly gained popularity due to their success in natural language processing and other domains, are well-suited to address these challenges due to their ability to model long-range dependencies and process data in parallel. But their application to EHR classification remains limited by data representations, which can reduce performance or fail to capture informative missingness. In this paper, we present the Bi-Axial Transformer (BAT), which attends to both the clinical variable and time point axes of EHR data to learn richer data relationships and address the difficulties of data sparsity. BAT achieves state-of-the-art performance on sepsis prediction and is competitive to top methods for mortality classification. In comparison to other transformers, BAT demonstrates increased robustness to data missingness, and learns unique sensor embeddings which can be used in transfer learning. Baseline models, which were previously located across multiple repositories or utilized deprecated libraries, were re-implemented with PyTorch and made available for reproduction and future benchmarking. Keywords Electronic Health Records, Predictive Models, Time Series Analysis 1 Introduction Electronic Health Records (EHRs) are being produced in increasing numbers around the world, with many countries incentivizing or requiring their use [ 1 , 2 , 3 ] . This digitalization of healthcare data has given researchers easier access to patient diagnoses, treatments, lab tests, and anthropometric measurements, which can be used to train machine learning models for a variety of clinical applications [ 4 , 5 , 6 ] . Classification tasks are popular with EHR data, including predictions of medical codes, hospital readmission, and mortality [ 7 , 8 , 9 ] . Many deep learning approaches utilizing EHRs for classification have framed them as time series. Whereas traditional time series data are regular and dense, EHR data are irregular across patients and times, highly sparse, multimodal, and subject to experimental and biological noise (Fig. 1 ) [ 10 , 11 , 12 ] . A variety of successful models have been developed to address one or more of these characteristics [ 13 ] . Figure 1: Measurements from five different sensors, generated from a single ICU visit in the Physionet 2012 dataset [ 14 ] . Each sensor demonstrates varying levels of sparsity and irregularity. Nevertheless, additional challenges are present, accompanying the release of datasets with large numbers of patients, increasingly long histories, and more modalities [ 15 , 16 , 17 ] . Modeling patients across multiple health events may require the merging of several datasets, and databases are incorporating records sourced from different health-care providers [ 18 , 19 ] . As the volume and dimensionality of EHR data grows, the limitations of existing architectures may become a bottleneck. In this study, we implement modifications to the transformer architecture to increase EHR classification performance and make it more suitable to the arising complexity of EHR datasets. Our model, the Bi-Axial Transformer (BAT), learns embeddings that can be utilized for time and sensor attention and preserves signal patterns, enabling it to model complex, multi-axis relationships and the utilize informative missingness, which we demonstrate with visualized attention weights. We quantify how the defining characteristics of EHR datasets, including sparsity, patient demographics, and observed signals, affect performance and give BAT an edge over other transformers. Finally, we look at how the sensor embeddings learned by BAT can be utilized to integrate multiple datasets. Notably, several of the baseline models are implemented with deprecated packages that prohibit a straightforward benchmarking. We re-implemented these models with PyTorch, increased their computational efficiency with a variety of optimizations, and provide them in an open-sourced repository together with BAT at https://github.com/gsn245/Bi-Axial_Attention_Transformer. A preliminary version of this work has been reported [ 20 ] . 2 Related Work A variety of deep learning architectures have been utilized for EHR classification, including Convolutional Neural Networks (CNN), Autoencoders, and Graph Neural Networks (GNN) [ 21 , 22 , 23 , 24 ] . Among the architectures used, recurrent neural networks (RNNs) are some of the most popular models, including variants utilizing Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which frequently achieve top performance [ 25 , 26 , 27 , 28 , 29 ] . These models are designed to handle sequential data, with an inductive bias towards recent data points and an ability to handle variable-length inputs. This makes them a natural fit for EHR data, which contains time series of varying lengths due to irregular sampling, and recent observations that may be closer to death or disease onset. But the nature of digital patient data is evolving, and emerging research is exploring the integration of multiple modalities, such as images and omics, and the harmonization of datasets [ 30 , 31 , 32 , 33 , 34 ] . RNNs often handle new modalities, such as patient metadata, either by projecting it into vector space as the initial hidden state, or by concatenating it to each hidden state before classification. This approach may be unsustainable for longer time series and large modalities, as the architecture is known to have difficulties handling longer input sequences [ 35 ] . These observations highlight the need for new models which can be utilized for the future of EHR classification. In recent years, EHR publications have increasingly utilized transformers [ 36 , 37 , 38 ] . Since the introduction of their architecture by [ 39 ] , transformers have been used to explore multidimensional data from novel perspectives, resulting in strong performance in applications such as computer vision and natural language processing (NLP) [ 39 , 40 ] . In comparison to RNNs, transformers show great promise for the future of EHR research, as they perform better on longer sequences, can be parallelized, and can learn richer contextual representations with multi-headed attention [ 41 , 42 , 36 ] . There are 2 popular categories of data representation that have been used by transformers for EHR data. The first type represents data points as independent observations in a dense format, often as tuples including the sensor type, time, and measured value. The second type embeds the data as a series of either time points or sensors, condensing multiple observations into a shared embedding that retains signal patterns. The first strategy is used by models such as SeFT and STraTs [ 43 , 38 ] , which are very efficient for highly sparse EHR datasets. However, their dense input representations render them unable to learn from potentially meaningful signal patterns known as informative missingness. Informative missingness occurs when the absence of data conveys meaningful information, often reflecting underlying clinical decisions or patient conditions in EHRs. For instance, certain laboratory tests may be performed only when a clinician suspects specific ailments, as highlighted in Beaulieu-Jones et al. [ 44 ] ’s study on structured EHR data. This implies that missing data are frequently not random but instead influenced by factors

Read the complete text