Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel, Lin Yue, Weitong Chen · 2026
Read Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey by Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel, Lin Yue, Weitong Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27431.txt.
Introduction
Liangwei Nathan Zheng and colleagues survey mixture-of-experts methods for multimodal learning tasks. They frame the approach three ways: as a fast engine splitting cost from size, as a representation learner fusing expert knowledge, and as an adapter for missing or imbalanced modalities. Gaps remain in clear routing, expert talk, and long-term multimodal learning.
Catalog record
| CREATOR | Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel, Lin Yue, Weitong Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36833 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27431
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel, Lin Yue, Weitong Chen Abstract Mixture-of-Experts (MoE) presents a naturally compatible and scalable framework for multimodal learning, demonstrating strong adaptability across diverse modalities and tasks. Despite its growing success, a comprehensive and systematic review on the MoE metho addressing multimodal challenges remains lacking. Existing surveys tend to evaluate either multimodal learning or MoE independently from method taxonomy, overlooking the unique interplay between them. This survey fills the gap by answering a central question: How does MoE effectively resolve multimodal challenges? We approach this from three key perspectives: (1) MoE as an Efficient Multimodal Engine: enabling scalable multimodal modeling by decoupling computational cost from parameter growth and mitigating modality redundancy through selective expert activation; (2) MoE as a Multimodal Representation Learner: integrating complementary multi-opinion expert knowledge to enrich alignment and interaction representations; and (3) MoE as a Multimodal Adapter: providing a modular and flexible mechanism to model imperfect data scenarios such as modality imbalance and missing modality. Through our extensive literature review, we identify critical research gaps, including interpretable routing, expert communication, modality integration, and lifelong multimodal learning. We position this survey as a foundation for future research toward interpretable and sustainable multimodal Mixture-of-Experts system. 1 Introduction Real-world data modeling is a mixing of various heterogeneous data, also termed as multimodal data, which brings comprehensive and multiple perspectives understanding of target (e.g. a complete patient representation can be viewed as the composition of EHR, medical image, lab test and so on.) ( 38 ; 32 ) . In recent year, many works have made successful progress on multimodal learning in many applications such as medical ( 44 ; 12 ) and generative AI ( 6 ; 39 ) . We recognize four main objectives of multimodal learning from our literature reviews: (1) Multimodal Model Scaling is an objective to efficiently scale the multimodal model and enable large learning capability without significant increase of computational cost. Multimodal dataset usually offers multiple modality perspective of one instance, which typically requires larger computational costs for one instance ( 17 ; 18 ) . (2) Fusion and Interaction is an objective to jointly learn the synthetic interaction when two or more modalities present in one instance. For example, combining visual and textual information for video sentiment analysis ensures that complementary signals are leveraged together. Such a fusion feature usually combines the unique information from each modality and generates an interacted feature from the combination ( 21 ; 40 ) . Figure 1: Types of Mixture-of-Experts for Multimodal Learning (3) Alignment and Translation is an objective to map heterogeneous modalities into a shared representation space such that semantically attributes are closely matched. A well-aligned modality also contribute to translate the modality under heterogeneous structure. This enables cross-modal understanding, for example aligning spoken words with lip movements in video. The alignment objective usually emphasizes semantic consistency across modalities while preserve their structural discrepancy with shared information between heterogeneous modality ( 28 ) . (4) Modality Robustness is an objective to build resilient modality representation under imperfect data stream, where the modality data can be missing, corrupted and incorrect. For example, severely missing chest ray modality for patient data while the EHR modality is often observed. It aims to build systems that exhibit dynamic resilience, ensuring that the model maintains a comprehensive understanding of the task despite fluctuations in data availability, quality, and the very composition of the modality stream ( 44 ; 11 ) . Many multimodal learning models recently are designed as dense and achieve acceptable performance. However, dense models face challenges on multimodal learning such as training inefficiency, poor generalization on learning multimodal heterogeneity structure and poor generalization on multi-task adaption ( 2 ) . To address achieve the four main challenges in multimodal learning, researchers found that Mixture-of-Experts (MoE) provides new learning paradigm that offers a natural solution to multimodal learning as shown in Figure 1 : (1) MoE can be sparsely activated to scale up the multimodal model without significant increase of computational cost. (2) By routing modality-specific inputs to experts, localized representation learning that preserves modality-unique characteristics while facilitating cross-modal interaction. This specialization mechanism allows MoE to disentangle modality-dependent features and recompose them into semantically consistent representations. (3) MoE enables a multi-opinion mechanism that enriches the translation process. Instead of relying on a single shared representation, each expert captures unique perspectives of modality correspondence. Despite the strong capability of MoE in multimodal learning, there still lack of a comprehensive literature reviews of relevant works to the best of our knowledge. There are some surveys discuss the multimodal learning ( 38 ; 7 ) and MoE architecture ( 27 ; 3 ) , but they somehow overlook that significance impacts and advantages of MoE for multimodal data and discuss only a few MoE methods in multimodal learning. In the light of this, we systematically study the most recent state-of-art (SOTA) methods as shown in Figure 2 . Our taxonomy discuss the main challenges in multimodal learning and the specific MoE solution to address them. We primarily discuss (1) Multimodal scaling approach by the Mixture-of-Width and -Depth as in Figure 1 . (2) Achieving multimodal alignment and interaction by expert specialized learning. (3) Addressing challenging multimodal problems by MoE such as missing modality and multimodal life-long learning. In addition, we further discuss current bottleneck and potential research directions for Multimodal MoE. We position this survey to bring complete views on process of using MoE for multimodal learning and to be specifically robust, so as to be an insightful guidebook for researchers who interest in multimodal MoE in this area. {forest} Figure 2: A taxonomy of approaches for multimodal learning with Mixture-of-Experts 2 Scope and Literature Collection Method This survey collected papers from January 2020 to October 2025 from the following venues: ICLR, ICML, NeurIPS, ACL, EMNLP, CVPR, ECCV, ICCV, AAAI, IJCAI, KDD, WWW, MICCAI, and some are technical reports, and insightful preprint paper. Note that this survey will not discuss the papers where MoE is used simply as the feature extractor or simple replacement of FFN layer of Transformer without or with little technically and innovatively contribution to multimodal learning. Method Modality Backbone Model MoE Type Task Domain Code CuMo ( 17 ) Image, Text LLM & CLIP Top-K MoWE Multi-Task General MoE-LLaVA ( 20 ) Image, Text LLM Top-K MoWE Multi-Task General Uni-MoE ( 18 ) Image, Text, Audio, Speech, Video LLM Top-K MoWE Multi-Task General Med-MoE ( 12 ) Image, Text LLM Top-K MoWE Multi-Task Medical Gamma-MoD ( 24 ) Image, Text LLM MoDE Multi-Task General UniMoD ( 26 ) Image, Text LLM MoDE Multi-Task General LIMoE ( 28 ) Image, Text Transformer Top-K MoWE Representation Learning General Eagle ( 31 ) Image, Text LLM Specialized Expert VQA General MedMoE ( 5 ) Multimodal Brain Slice Transformer Specialized Expert Representation Learning Medical N/A IMP-MoE ( 1 ) Image, Spectrogram, Text, Waveform Transformer Top-K MoWE Representation Learning General N/A MoMoK ( 43 ) Image, Text, Knowledge Graph Variational Network GNN Specialized Expert Representation Learning Knowledge Graph RAPHAEL ( 39 ) Image, Text Diffusion Time Step Expert Image Generation General N/A R2-T2 ( 19 ) Image, Text LLM