Fusion or Confusion? Multimodal Complexity Is Not All You Need

Tillmann Rheude, Roland Eils, Benjamin Wild · 2025

Read Fusion or Confusion? Multimodal Complexity Is Not All You Need by Tillmann Rheude, Roland Eils, Benjamin Wild (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22991.txt.

Introduction

Tillmann Rheude, Roland Eils, and Benjamin Wild question whether fancier multimodal designs actually fuse information better. Multimodal learning promises gains from combining data types, and method papers have grown steadily more complex on the belief that specialized fusion machinery pays off. The authors rebuilt nineteen high-impact multimodal methods and ran them across nine datasets with up to twenty-three modalities under one shared protocol.

With tuning, setup, cross-validation, and statistical testing held uniform, added complexity often produced confusion rather than fusion. Complex designs did not reliably beat unimodal baselines or the authors' deliberately simple baseline for multimodal learning. A focused case study then exposes basic shortcomings in even top-tier publications, such as uneven tuning budgets. The conclusion is a plea to shift effort from architectural novelty toward rigor in testing, so the field's comparisons can be trusted.

Catalog record

CREATORTillmann Rheude, Roland Eils, Benjamin Wild
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fusion or Confusion? Multimodal Complexity Is Not All You Need Tillmann Rheude, Roland Eils, Benjamin Wild Abstract Deep learning architectures for multimodal learning have increased in complexity, driven by the assumption that multimodal-specific methods improve performance. We challenge this assumption through a large-scale empirical study reimplementing 19 19 high-impact methods under standardized conditions, evaluating them across nine diverse datasets with up to 23 23 modalities, and testing their generalizability to new tasks beyond their original scope, including settings with missing modalities. We propose a Sim ple Ba seline for M ulti m odal Learning ( SimBaMM ), a straightforward late-fusion Transformer architecture, and demonstrate that under standardized experimental conditions with rigorous hyperparameter tuning of all methods, more complex architectures do not reliably outperform SimBaMM . Statistical analysis indicates that more complex methods perform comparably to SimBaMM and frequently do not reliably outperform well-tuned unimodal baselines, especially in the small-data regime considered in many original studies. To support our findings, we include a case study of a recent multimodal learning method highlighting the methodological shortcomings in the literature. In addition, we provide a pragmatic reliability checklist to promote comparable, robust, and trustworthy future evaluations. In summary, we argue for a shift in focus: away from the pursuit of architectural novelty and toward methodological rigor. 4 4 4 The code repository is available on GitHub . 1 Introduction The pursuit of state-of-the-art (SoTA) performance in multimodal learning has led to increasingly complex architectures. The field is engaged in an architectural arms race, driven by the implicit assumption that novel fusion mechanisms, gradient-based regularization, and deeper, more sophisticated networks are the primary drivers of progress. This has resulted in a landscape in which models are often celebrated for their structural novelty, with performance gains on specific benchmarks presented as evidence of their superiority. However, such gains are difficult to interpret when evaluation protocols differ across papers and when key experimental factors , e.g., hyperparameter tuning, weight initialization, and missing-modality simulation, are not consistently controlled. In this paper, we challenge this prevailing narrative that architectural novelty is the primary driver of performance in multimodal learning. We argue that methodological rigor in model selection and evaluation can explain a substantial portion of reported improvements. Accordingly, we hypothesize that performance gains often attributed to complex design choices may not reliably persist under standardized experimental conditions. To substantiate this claim, we perform a large-scale empirical study evaluating 19 multimodal architectures across nine datasets with up to 488\,131 samples and up to 23 modalities. Our goal is to disentangle the contributions of architectural complexity from those of experimental practice. Our findings reveal that increased architectural complexity does not yield consistent gains over a simple late-fusion baseline when all methods are tuned and evaluated under the same rigorous protocol. Our contributions are summarized as follows: Figure 1 : Taxonomy of multimodal learning methods exemplified by two modalities. A late-fusion baseline ( SimBaMM ) processes a multimodal dataset D with up to M modalities through encoders E_{m} , combines them via fusion, and produces predictions through a classification head. Categorized into five groups, we evaluate 19 SoTA methods across nine datasets based on their architectural innovations on top of SimBaMM : Missing Data handling for incomplete modalities, Encoder -level modifications, Fusion mechanisms for combining representations, Head -based cross-modal architectures, and Gradient -based optimization techniques. Our large-scale empirical study reveals that none of these complex extensions reliably outperforms SimBaMM , suggesting that reported performance gains are often attributable to experimental confounders rather than methodological novelty. • A large-scale benchmark of multimodal architectures under a unified protocol We re-implement and evaluate 19 multimodal methods across nine diverse datasets, enabling controlled comparisons and assessing how well performance transfers beyond the original evaluation settings. • A Simple Baseline for Multimodal Learning (SimBaMM) We introduce SimBaMM , a late-fusion Transformer baseline that handles missing modalities through masking ( Figure 1 ), and show that it matches or exceeds the performance of more complex architectures in many settings under rigorous tuning and standardized evaluation. • A case study revealing how experimental confounders inflate reported performance We demonstrate weaknesses in existing complex multimodal models, such as information leakage during validation and inadequate hyperparameter tuning, by including a case study of one recent multimodal learning method. 2 Related Work Multimodal Learning Research in multimodal learning addresses the use of multiple modalities to perform prediction tasks. We focus on architectural contributions to multimodal representation learning rather than multimodality-specific applications. Architectures in multimodal deep learning can be divided into groups ( Figure 1 ) spanning missing-data handling (see next paragraph), encoder developments ( Ghahremani & Wachinger 2023 ) , fusion mechanisms ( Nagrani et al. 2021 ; Kontras et al. 2025 ; Liu et al. 2018 ; Cao et al. 2024 ; Li et al. 2024 ; Wu et al. 2025 ) , head models ( Tsai et al. 2019 ; Jiang et al. 2025a ) , and gradient regulation methods ( Peng et al. 2022 ; Wei et al. 2025a ; Wei et al. 2025b ; Wei & Hu 2024 ; Wu et al. 2022 ; Wang et al. 2020 ) . There is also work on meta-learning ( Ma et al. 2021 ) and similar extensions such as pre-training or distillation ( Li et al. 2021 ) to multimodal learning, which are excluded in our work. Recent work conducted a large-scale study of scaling laws for native multimodal models and compared mainly early- and late-fusion designs ( Shukor et al. 2025 ) . This line of work is complementary to ours: it focuses on scaling trends within a restricted set of architectural families, whereas we benchmark a broader set of contemporary multimodal methods across diverse datasets and emphasize statistically rigorous evaluation and model selection. Furthermore, our work is situated within the context of large-scale multimodal benchmarking, with MultiBench ( Liang et al. 2021 ) representing the most closely related effort. However, we extend and differentiate our work from this foundation and the aforementioned work in several key ways. First, we focus exclusively on a curated set of large-scale, contemporary, real-world datasets to test the generalizability of more modern architectures. Second, we directly assess cross-domain performance rather than separating in-domain and out-of-domain analyses, to attribute performance gains to methodological contributions and to test whether these gains generalize across domains ( Lipton & Steinhardt 2019 ) . Third, to ensure a rigorous and fair comparison, we standardize crucial implementation details like weight initialization and release our implementation publicly. Finally, we introduce SimBaMM that addresses the reliability aspect of multimodal learning that was previously underexplored in the literature. In contrast to a general benchmark such as MultiBench, our work is a cohesive comparative study that establishes a strong, up-to-date baseline for the field. In summary, we do not seek to advance multimodal architectures but to test whether recent methodological advances withstand rigorous evaluation. Multimodal Learning with Missing Modalities Prior work on robustly handling missing modalities is broadly classified into two main categories: strategy design (which includes architecture-focused designs and model combinations) and data processing (which involves representation learning and modality imputation) ( Wu et al. 2024 ) . If missing modalities are not imputed directly at the data level, prior knowledge

Read the complete text