CTPD : Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis

Fuying Wang, Feng Wu, Yihan Tang, Lequan Yu · 2024

Read CTPD : Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis by Fuying Wang, Feng Wu, Yihan Tang, Lequan Yu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00696.txt.

Description

Abstract Integrating multimodal Electronic Health Records (EHR) data—such as numerical time series and free-text clinical reports—has great potential in predicting clinical outcomes. However, prior work has primarily focused on capturing temporal interactions within individual samples and fusing multimodal information, overlooking critical temporal patterns across patients.

Catalog record

CREATORFuying Wang, Feng Wu, Yihan Tang, Lequan Yu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CTPD : Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis Fuying Wang, Feng Wu, Yihan Tang, Lequan Yu Abstract Integrating multimodal Electronic Health Records (EHR) data—such as numerical time series and free-text clinical reports—has great potential in predicting clinical outcomes. However, prior work has primarily focused on capturing temporal interactions within individual samples and fusing multimodal information, overlooking critical temporal patterns across patients. These patterns, such as trends in vital signs like abnormal heart rate or blood pressure, can indicate deteriorating health or an impending critical event. Similarly, clinical notes often contain textual descriptions that reflect these patterns. Identifying corresponding temporal patterns across different modalities is crucial for improving the accuracy of clinical outcome predictions, yet it remains a challenging task. To address this gap, we introduce a C ross-Modal T emporal P attern D iscovery ( CTPD ) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data. Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings. To ensure rich cross-modal temporal semantics in the learned patterns, we introduce a contrastive-based TPNCE loss for cross-modal alignment, along with two reconstruction losses to retain core information of each modality. Evaluations on two clinically critical tasks—48-hour in-hospital mortality and 24-hour phenotype classification—using the MIMIC-III database demonstrate the superiority of our method over existing approaches. $*$ $*$ footnotetext: F. Wang and F. Wu contributed equally to this work. 1 Introduction The increasing availability of Electronic Health Records (EHR) presents significant opportunities for advancing predictive modeling in healthcare ( Acosta et al. 2022 ; Wang et al. 2024 ) . EHR data is inherently multimodal and time-aware, encompassing structured data like vital signs, laboratory results, and medications, as well as unstructured data such as free-text clinical reports Kim et al. 2023 . Integrating these diverse data types is crucial for comprehensive patient monitoring and accurate prediction of clinical outcomes Hayat et al. 2022 ; Wang et al. 2023 ; Zhang et al. 2023 . However, the irregularity and heterogeneity of multimodal data present significant challenges for precise outcome prediction. Figure 1: Motivation of our proposed CTPD : The core idea is to extract cross-modal temporal patterns (e.g., abnormal trends in blood pressure) to enhance the clinical outcome prediction from multimodal EHR data. Existing approaches primarily address either the irregularity of data Shukla and Marlin 2019 ; Horn et al. 2020 ; Zhang et al. 2021 ; Zhang et al. 2023 or the fusion of multiple modalities Huang et al. 2020 ; Zhang et al. 2020b ; Xu et al. 2021 ; Kline et al. 2022 , but they often neglect broader temporal trends that span across patient cases. These cross-modal temporal patterns, present in both structured and unstructured data, can provide high-level semantic insights into a patient’s health trajectory and potential risks Conrad et al. 2018 . For instance, worsening trends in the vital signs of a patient, such as excessively high heart rate or blood pressure, may signal deteriorating health or a critical event, and these trends can also be reflected in clinical notes. Learning these global patterns in a cross-modal, temporal manner is expected to improve predictive performance. Furthermore, existing methods struggle to capture temporal patterns across varying time scales, which is critical for robust time series analysis. To address these limitations, we propose the C ross-modal T emporal P attern D iscovery ( CTPD ) framework, designed to extract meaningful temporal patterns from multimodal EHR data to improve the accuracy of clinical outcome predictions. As shown in Fig. 1 , the core innovation of our approach is a novel temporal pattern discovery module, which identifies corresponding temporal patterns (i.e., temporal prototypes) with meaningful semantics across both modalities throughout the dataset. This approach ensures that the model captures essential temporal semantics relevant to patient outcomes, providing a more comprehensive understanding of the data. To further enhance the quality of the learned temporal patterns, we introduce a InfoNCE Oord et al. 2018 -based TPNCE loss for aligning pattern embeddings across modalities, along with auxiliary reconstruction losses to ensure that the patterns retain core information of the data. Moreover, our framework incorporates a transformer Vaswani et al. 2017 -based fusion mechanism to effectively fuse the discovered temporal patterns with timestamp-level representations from both modalities, leading to more accurate predictions. We evaluate CTPD on two critical clinical prediction tasks: 48-hour in-hospital mortality prediction and 24-hour phenotype classification, using the MIMIC-III database. The results demonstrate the effectiveness of our approach, which significantly outperforms existing methods, and suggest a promising direction for improving multimodal EHR analysis for clinical prediction. Contributions: (i) . We propose a novel framework that captures cross-modal temporal patterns across multiple time scales in multimodal EHR data, offering a more comprehensive understanding of longitudinal patient information. (ii) . Our framework incorporates a temporal pattern discovery module, utilizing slot attention to effectively extract meaningful semantic information. (iii) . We introduce a TPNCE learning objective to optimize the temporal pattern discovery process, ensuring richer and more informative pattern learning. (iv) . Our framework outperforms existing methods in two clinically critical tasks, demonstrating superior predictive performance. 2 Related Works 2.1 EHR Time-Series Data Analysis EHR data is critical for clinical tasks such as disease diagnosis, mortality prediction, and treatment planning ( Harutyunyan et al. 2019 ; Zhang et al. 2023 ) . However, its high dimensionality and irregular nature pose challenges for traditional predictive models ( Rani and Sikka 2012 ; Lee et al. 2017 ) . Deep learning models, such as RNNs and LSTMs, are often used to capture temporal dependencies in EHR data ( Hayat et al. 2022 ; Deldari et al. 2023 ) , but they struggle with irregular time intervals due to their reliance on fixed-length sequences ( Xie et al. 2021 ) . To address this, some methods update patient representations at each time step using graph neural networks ( Zhang et al. 2021 ) , while others employ time-aware embeddings to incorporate temporal information ( Qian et al. 2023 ; Zhang et al. 2023 ) . Despite these advancements, existing approaches still fail to capture high-level temporal patterns that are crucial for clinical outcomes. Subtle interactions between clinical parameters over extended periods—such as the relationship between blood pressure and creatinine levels—often hold the key to accurate predictions ( Hanratty et al. 2011 ; Sun et al. 2024 ) . Moreover, current methods struggle to model temporal patterns across varying time scales, limiting their effectiveness in clinical outcome predictions. 2.2 Prototype-based Pattern Learning Prototype-based learning identifies representative instances (prototypes) and optimizes their distance from input data in latent space for tasks like classification ( Li et al. 2023a ; Ye et al. 2024 ) . This approach has been widely applied in tasks such as anomaly detection, unsupervised learning, and few-shot learning ( Tanwisuth et al. 2021 ; Li et al. 2023b ) . Recently, it has been extended to time-series data ( Ghosal and Abbasi-Asl 2021 ; Li et al. 2023a ; Yu et al. 2024 ) , demonstrating its potential for detecting complex temporal patterns. Additionally, prototype-based learning offers interpretable predictions, which is essential for healthcare applications ( Zhang et al. 2024 )

Read the complete text