Challenges and proposed solutions in modeling multimodal medical data: A systematic review
Maryam Farhadizadeh, Maria Weymann, Michael Blaß, Johann Kraus, Christopher Gundler, Sebastian Walter, Noah Hempen, Hannah Bast, Harald Binder, Nadine Binder · 2025
Read Challenges and proposed solutions in modeling multimodal medical data: A systematic review by Maryam Farhadizadeh, Maria Weymann, Michael Blaß, Johann Kraus, Christopher Gundler, Sebastian Walter, Noah Hempen, Hannah Bast, Harald Binder, Nadine Binder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06945.txt.
Introduction
Maryam Farhadizadeh and nine co-authors reviewed the snags in modeling mixed-kind medical data, in a paper published in 2025. Mixed sets join scans, genes, wrist sensors, and chart files, and can sharpen decisions and tailor care. Their sweep kept 69 studies and sorted the blocks they hit. Five snags recurred: missing kinds in some files, small sets, size clashes among kinds, unclear cause readings, and the fusion craft itself.
Fixes on the rise span transfer learning, synthetic data, attention weights, and automatic build searches. The team maps which fix meets which snag. They aim the map at builders of joint models for clinics. The next step for the field, they say, is fitting craft, not more data kinds.
Catalog record
| CREATOR | Maryam Farhadizadeh, Maria Weymann, Michael Blaß, Johann Kraus, Christopher Gundler, Sebastian Walter, Noah Hempen, Hannah Bast, Harald Binder, Nadine Binder |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06945
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Challenges and proposed solutions in modeling multimodal medical data: A systematic review Maryam Farhadizadeh, Maria Weymann, Michael Blaß, Johann Kraus, Christopher Gundler, Sebastian Walter, Noah Hempen, Hannah Bast, Harald Binder, Nadine Binder Abstract Background: Multimodal data analysis has gained significant attention in biomedical Artificial Intelligence. Integrating diverse data sources, such as omics, wearable sensors, imaging, environmental, and electronic health records, offers a more comprehensive view of patient health and supports personalized care. However, modeling multimodal data poses several challenges that must be addressed to ensure accurate and reliable analysis. This systematic review aims to identify and analyze the challenges encountered in modeling multimodal data in clinical domains. Methods: A systematic search of PubMed was conducted to identify original studies published between 2011 and October 2023 that modeled two or more data modalities for medical applications and explicitly addressed at least one modeling challenge. In addition, a supplementary search was performed in Google Scholar to capture further relevant studies. Studies were included if they addressed the use of multimodal data in clinical settings, focusing on data from diverse modalities. Papers that focused solely on multimodal imaging or included audio and video data were excluded during the title and abstract screening. A full-text review of 84 papers was then performed, leading to additional exclusions for studies that did not directly address modeling challenges specific to multimodal data in the clinical domain. Data extraction was performed to identify reported challenges along with their corresponding solutions or proposed approaches. Results: This review included 69 studies (58 from the systematic search and 11 identified through a supplementary Google Scholar search). Five major categories of challenges were identified: missing modalities, imbalance in dimensionality, optimal fusion strategy, small datasets, and interpretability. Small datasets (17/69, 25%) and missing modalities (15/69, 22%) were the most frequently addressed challenges. Intermediate fusion was the most commonly used fusion strategy (39/69 studies). Missing modalities were addressed using techniques such as multitask learning, matrix completion, and generative adversarial networks. Small datasets were commonly tackled through data augmentation, transfer learning, and knowledge distillation. Interpretability was addressed using attention mechanisms and gradient-based attribution methods, while neural architecture search was explored to optimize fusion strategies. Conclusions: Addressing the main challenges of multimodal modeling in clinical domains, this review summarizes emerging strategies and synthesizes current trends and methodological advances. It provides a comprehensive overview of the field, highlights opportunities for future research and development, and aims to support the design of more robust multimodal models and improve data integration for clinical applications. keywords Multimodal data, Challenges, Fusion strategy, Modeling, Clinical applications, Deep learning Background In healthcare, data comes in various modalities like electronic health records (EHR), imaging, genomics, and wearable devices. Integrating more than one of these modalities in a multimodal data analysis holds significant promise for improving our understanding and management of complex health conditions Mohsen et al. [2022] , Baltrušaitis et al. [2019] . One notable domain where multimodal data analysis can make a transformative impact is in the management of chronic diseases such as asthma and chronic obstructive pulmonary disease (COPD) Acosta et al. [2022] . Here, medical imaging data, such as computed tomography (CT), provides insights into lung morphology and pathological changes. Wearable devices monitor physiological parameters and activity levels, offering real-time data on the patient’s disease trajectory. Environmental data, encompassing air quality and pollutants, helps identify external factors influencing respiratory conditions. Clinical records and genomic information contribute to personalized treatment strategies, while patient-reported outcomes capture subjective experiences, shaping a patient-centric approach. Combining these complementary modalities provides a more comprehensive understanding of disease and enables the identification of interactions between different aspects of patient health. We use this example to illustrate the general potential of multimodal integration in chronic disease management; as discussed in the Limitations, the studies identified in our systematic search were concentrated predominantly in neurology and oncology rather than in COPD or asthma specifically. Recent bibliometric analyses further demonstrate the rapidly growing interest in multimodal Artificial Intelligence (AI) for healthcare, highlighting multimodal data fusion as an increasingly active research area with applications spanning diagnosis, prognosis, and clinical decision support Chen et al. [2024] . Despite this rapid progress, modeling multimodal data remains a challenging task that extends well beyond the analysis of individual modalities because individual modalities differ in their data types, formats, scales, levels of noise, and interpretability. Moreover, modalities may exhibit varying degrees of completeness, sparsity, and dimensionality, making it challenging to integrate and analyze them together. Addressing these challenges requires effective multimodal fusion strategies, which combine information from different modalities into a unified representation. Learning such shared representations across heterogeneous modalities has been a central objective of multimodal deep learning since its early development Ngiam et al. [2011] . Fusion strategies can be broadly divided into four main categories: (i) early fusion, (ii) intermediate fusion, (iii) late fusion, and (iv) hybrid fusion. (i) Early fusion involves integrating information from different sources at the data level without learning any marginal representations or feature extraction, treating them as unimodal input. Early fusion offers simplicity by integrating modalities at the data level, however, it often assumes that all input modalities are directly compatible and equally informative. It can capture basic cross-modal relationships present in raw data, but it typically struggles to model more complex, high-level interactions between modalities. In particular, early fusion may be inadequate when modalities differ in dimensionality, sampling rates, or semantic content, such as combining image data with structured omics data, where aligning and weighting heterogeneous features is non-trivial. (ii) Intermediate fusion, in contrast, involves learning modality-specific representations to identify correlations within each modality before generating joint representations or making predictions. This approach offers flexibility in selecting different architectures for feature extraction from various modalities Stahlschmidt et al. [2022] . By employing distinct architectures and layers for each modality, it can effectively handle heterogeneous data. Additionally, intermediate fusion is resilient to imbalance in dimensionality Stahlschmidt et al. [2022] and can accommodate missing modalities without significantly affecting performance Stahlschmidt et al. [2022] , Lee and Van der Schaar [2021] . (iii) Late fusion, on the other hand, trains separate models for each modality and integrates their individual predictions at a later stage. This method is particularly well-suited for managing heterogeneous modalities and addressing imbalance in dimensionality and missing data. However, it may be less effective for modalities with strong correlations, as it does not capture interactions between different data sources. In addition, in scenarios where specific information holds greater significance, using late fusion offers notable benefits. The model prioritizes the most important information it has, then adds more details from other sources to improve its understanding. This prioritization allows the model to integrate complementary information from multiple modalities while emphasizing the most informative sources, thereby enhancing decision-making and prediction performance. (iv) Hybrid fusion combines elements from the aforementioned methods, allowing for flexible processing across multiple stages. This approach is especially effective in scenarios with diverse data modalities, where early fusion can be used for closely related data types, and later fusion stages facilitate the integration of more distinct modalities. It provides the adaptability needed to meet the specific requirements of multimodal models. Figure 1 illustrates how diverse data modalities can be integrated within a multimodal modeling framework. It