Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling
Andrew Wang, Ellie Pavlick, Ritambhara Singh · 2026
Read Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling by Andrew Wang, Ellie Pavlick, Ritambhara Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18753.txt.
Introduction
Wang, Pavlick, and Singh treat clinical diagnosis as sequence modeling to handle missing data types in patient trajectories. Patient data arrive over time and often lack some data types, so the team first builds a contrastive pretraining objective. It is aware of missingness and places all data types in one shared latent space. Sequence models built on transformers then beat baselines on MIMIC-IV and eICU benchmarks. Analysis shows that dropping data types changes model behavior in ways that diverge, a problem the training mitigates.
Catalog record
| CREATOR | Andrew Wang, Ellie Pavlick, Ritambhara Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71059 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18753
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling Andrew Wang, Ellie Pavlick, Ritambhara Singh Abstract An active challenge in developing multimodal machine learning (ML) models for healthcare is handling missing modalities during training and deployment. As clinical datasets are inherently temporal and sparse in terms of modality presence, capturing the underlying predictive signal via diagnostic multimodal ML models while retaining model explainability remains an ongoing challenge. In this work, we address this by re-framing clinical diagnosis as an autoregressive sequence modeling task, utilizing causal decoders from large language models (LLMs) to model a patient’s multimodal trajectory. We first introduce a missingness-aware contrastive pre-training objective that integrates multiple modalities in datasets with missingness in a shared latent space. We then show that autoregressive sequence modeling with transformer-based architectures outperforms baselines on the MIMIC-IV and eICU fine-tuning benchmarks. Finally, we use interpretability techniques to move beyond performance boosts and find that across various patient stays, removing modalities leads to divergent behavior that our contrastive pre-training mitigates. By abstracting clinical diagnosis as sequence modeling and interpreting patient stay trajectories, we develop a framework to profile and handle missing modalities while addressing the canonical desideratum of safe, transparent clinical AI. K eywords MIMIC-IV, eICU, Multimodal Machine Learning, Data Integration, Contrastive Learning, Interpretability 1 Introduction A patient’s clinical trajectory is inherently sequential, multimodal, and irregular in terms of information availability. Their diagnosis from a human clinician, therefore, does not always occur as a single prediction at the end of their observed stay, but often as a sequence of updated beliefs given the dynamic nature of the clinical context. Despite this, a dominant approach in multimodal clinical ML relies on static architectures that concatenate all patient data from a predefined window into a single vector ( 9 ). This "bag-of-features" approach enforces a static prediction schema, preventing clinicians from auditing the sequence of evidence that drives the model’s decision. Furthermore, existing approaches to handling missing modalities in these architectures typically fall into two paradigms: either inferring or imputing missing data that has not been clinically collected ( 19 , 28 ), or relying on uninformative zero-padding and sample-dropping ( 18 ). These methods either hallucinate clinical signals or destroy the underlying geometry of the latent space, both of which are detrimental to safe diagnostic modeling. To bridge the gap between clinical reality and ML methodology, we re-frame clinical prediction from a static classification task to the autoregressive modeling of belief states, re-purposing the established machinery of transformer-based 39 language model decoders. To ensure this temporal decoder can process sparse, multimodal data, we first propose a novel contrastive pre-training step. Using a hybrid objective function we term “Masked Global Alignment", we map all available modalities and explicitly learned placeholder tokens for missing modalities into a stable global representation of patients. Next, we use these pretrained embeddings to fine-tune and evaluate downstream models on both the MIMIC-IV ( 15 )/CXR ( 16 ) and eICU ( 29 ) datasets, demonstrating performance gains of sequential models over static baselines. Finally, we conduct a mechanistic analysis of the model’s temporal dynamics, illustrating the ability to transparently and mechanistically interpret the diagnostic reasoning process of autoregressive clinical sequence models. In doing so, we also demonstrate the utility of our contrastive pre-training approach, mitigating potential sources of bias by robustly encoding and aligning latent patient data. Taken together, our results suggest that a framework combining missingness-aware contrastive pre-training and transformer-based sequence modeling not only serves to mitigate missing-modality-associated ML performance loss but also to retain model explainability, which is crucial to trustworthy deployment and application. Figure 1: Overview of our proposed two-part missingness-aware framework. (a) We first develop a novel, missing modality-aware contrastive pretraining method, ensuring that the latent representations of the sparse data are aligned in a robustness manner. (b) We then re-purpose the machinery of autoregressive transformer-based models to model the sequence of contrastively pretrained embeddings for each patient stay, mimicking a realistic clinical diagnostic process and enabling more transparency into the sequential nature of decision making and updating. To summarize, our contributions are as follows: • We introduce a novel, hybrid contrastive loss that learns explicit missingness tokens, aligning incomplete multimodal patient states into a global latent representation that benefits models trained on datasets with a high degree of missing modalities. • We fuse the contrastive pretraining setup with transformer-based architectures to sequentially model patient trajectories, and show superior performance of sequential models over static baselines in the MIMIC-IV and eICU datasets. • We provide mechanistic evidence that contrastive alignment prevents unstable safety failures under missing modality conditions, and trace the reasoning trail of sequential transformer-based architectures to highlight the transparency enabled by clinical sequence modeling. 2 Related Work Building Multimodal Clinical Models Using MIMIC-IV and eICU : With regards to the MIMIC dataset, 2 , 33 , 48 , 5 , and 23 represent a subset of existing work that integrate demographics, lab results, chest X-Ray images, and other modalities to build predictive multimodal models. In the context of applications in healthcare using the MIMIC dataset, 49 covers the difficulties of debiasing multimodal representations on the MIMIC dataset, while 40 presents a multimodal approach to an applied problem in handling contextual confounders in ML model evaluation, developing contextual models of future CXR labels using prior discharge summaries. For the eICU dataset, existing work building multimodal clinical ML models includes 35 , which integrates demographic, laboratory measurements, and nurse bedside documentation, and 41 integrates medical codes, multivariate time series, clinical text, and demographics. Additionally, 46 and 32 notably provide methods that generalize to both the MIMIC-eICU datasets. Finally, 9 implements a similar contrastive pre-training pipeline to our study, utilizing modality-specific encoders in preparation for downstream task fine-tuning on the MIMIC-IV dataset to overcome performance degradation as modalities are removed from the evaluation set, although these approaches all make the assumption that all modalities are present in the training/evaluation sets. Handling Missing Modalities : In building predictive multimodal clinical ML models, existing approaches include dynamic Mixture of Experts models to address missing modalities on the fly 44 , transformer-based models to find correlations and representation alignment across potentially missing modalities in 24 , 38 , and 18 exploring different modality fusion strategies to handle this missingness. For more transparency in model prioritization, 47 disentangles features specific to EHRs and general across EHRs and images in a multimodal framework, producing disease- and patient-specific weights from an attention layer to dynamically weight each modality during prediction. In uncertainty quantification, 10 explicitly quantifies uni-modal aleatoric uncertainty to address missing modalities, resulting in significant performance gains on tasks in the MIMIC and eICU datasets. Other approaches in existing work include reconstructing missing modalities at inference time to mitigate performance degradation when these modalities are absent post-training ( 8 ), handling missing modalities by anchoring everything into the strongest, richest modality, using a multimodal transformer to handle correlations within and between modalities ( 26 ), and dynamic mixture of experts models with comparative training in learning process to directly regularize the geometry of the loss landscape directly ( 20 ). Additionally, 42 and 43 fuse shared features across modalities to enable modality-aware training and identify modality-specific features. However, these approaches