Soft Modality-Guided Expert Specialization in MoE-VLMs
Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Jiale Yan, Mo Guang, Kaiwen Long · 2026
Read Soft Modality-Guided Expert Specialization in MoE-VLMs by Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Jiale Yan, Mo Guang, Kaiwen Long (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23996.txt.
Introduction
Bo and eight colleagues propose SMoES, a routing scheme that lets vision-language mixture-of-experts models specialize experts by modality. Soft scores track how modalities fuse at each layer, experts are grouped for parallel deployment, and a regularization term keeps specialization coherent across groups. Across four models and 16 benchmarks it improves accuracy, cuts expert-parallel communication by more than half, and raises throughput.
Catalog record
| CREATOR | Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Jiale Yan, Mo Guang, Kaiwen Long |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75038 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soft Modality-Guided Expert Specialization in MoE-VLMs Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Jiale Yan, Mo Guang, Kaiwen Long Abstract Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency. 1 Introduction The Mixture-of-Experts (MoE) architecture 45 ; 27 ; 14 ; 18 has emerged as a cornerstone for modern large vision-language models (VLMs). Its principle of conditional computation allows for a massive expansion of model capacity with only a modest increase in the per-token computational budget by routing inputs to specialized expert sub-networks 45 ; 26 . This paradigm is particularly well-suited for fusing heterogeneous modalities. Consequently, leading systems like DeepSeek-VL2 60 , Kimi-VL 51 , GLM-4.5V 22 , and InternVL-3.5 55 have adopted MoE to achieve better performance while maintaining tractable inference costs. Despite this widespread adoption, a fundamental question remains under-explored: how do modality-specific signals (vision vs. text) interact with the expert routing mechanism, and can it be optimized to enhance both effectiveness and efficiency? Figure 1 : Comparison of routing strategies in MoE-VLMs. Hard routing enforces strict modality-expert separation, while soft routing allows arbitrary, uncontrolled mixing. SMoES uses soft modality scores to encourage dynamic expert specialization. Mainstream MoE-VLMs route tokens using either hard or soft routing paradigms ( Fig. 1 ). Hard routing 2 ; 56 pre-assigns experts to a specific modality, creating sharp specialization at the cost of rigid boundaries that hinder adaptation to cross-modal features and ignore the natural blending of representations across layers. In contrast, soft routing 42 ; 60 ; 51 ; 22 , the prevailing paradigm, allows experts to process any token. Yet, these methods often rely on heuristic priors or auxiliary losses disconnected from the evolving modality distributions, resulting in either over-mixing or under-specialization. Hybrid approaches 2 ; 56 ; 53 partition experts into modality-specific (hard-routed) and shared (soft-routed) groups, but this partitioning is typically hand-crafted and layer-agnostic, failing to align with feature evolution. Fundamentally, these paradigms are guided by idealized assumptions or uncontrolled mixing rather than the dynamic, data-driven geometry of modality distributions, offering little guidance on how expert specialization should evolve with network depth. We analyze modality fusion patterns of token features across layers in LLaVA-1.5 35 and a DeepSeekMoE-based 11 VLM, as shown in Fig. 2 . The analysis shows that modality fusion in MoE-VLMs is highly heterogeneous, with no clear boundary between modality-specific and cross-modal states. At a macro level, fusion patterns vary across models and layers, as reflected by distinct Jensen–Shannon (JS) divergence trajectories. At a micro level, even within the same layer and modality, some tokens remain modality-specific while others become cross-modal. This multi-scale heterogeneity indicates that rigid routing assumptions—either enforcing hard separation or imposing uniform mixing—are misaligned with how modalities actually interact across depth. Therefore, guided by signals that respect the evolving modality structure, the expert pool in MoE has the potential to naturally enable dynamic modality specialization. Figure 2 : Modality fusion pattern. Token feature distributions vary across models and layers. Text tokens are divided into several semantic clusters, while vision tokens are roughly separated into content and padding clusters. The last column shows the JS divergence between vision and text tokens across all layers. Beyond effectiveness, modality specialization is also closely tied to the inference efficiency of MoE-VLMs. Vision and text tokens differ sharply in both quantity and information density. Vision tokens often dominate the sequence length but typically carry lower information density due to spatial redundancy, whereas text tokens are fewer yet more semantically concentrated 5 ; 24 ; 40 . In real-world scenarios, inputs may even be text-only. This asymmetry poses two efficiency challenges. First, standard routing objectives coupled with vanilla load balancing tend to allocate most experts to the dominant yet low-information-density vision modality, hindering optimal specialization. Second, under expert parallelism (EP) 34 —a common deployment strategy—modality-agnostic routing scatters tokens across devices, inflating inter-communication overhead. Carefully-designed modality specialization addresses both issues: by establishing clear expert–modality affinities and scheduling tokens accordingly, aligned experts can be co-located on the same device, reducing communication while preserving balanced computation and capacity. Motivated by these observations, we propose Soft Modality-guided Expert Specialization ( SMoES ), which addresses these challenges through three key components. First, we introduce dynamic soft modality scores through two complementary estimators—attention-accumulated and Gaussian-statistics—that enable adaptive expert selection aligned with actual fusion states rather than hard boundaries. Second, we introduce expert binning—partitioning experts into coherent groups—which serves as the structural foundation for modality specialization and enables natural device placement units for modality-aware expert parallelism deployment. Third, to drive effective modality specialization, we introduce a mutual information (MI) objective that encourages different expert bins to specialize on distinct modality patterns. In summary, we make three key contributions: (1) Dynamic soft modality scores—Gaussian-statistics and attention-accumulated estimators—that align routing with the evolving modality fusion patterns across layers. (2) An expert binning mechanism coupled with an MI-driven specialization objective that promotes coherent modality grouping for both effectiveness and deployment efficiency. (3) Extensive experiments across four MoE-VLM backbones and 16 benchmarks demonstrating SOTA performance in accuracy and efficiency, including reduced EP communication overhead in realistic deployment. 2 Related Works Figure 3 : Overview of SMoES . Soft Modality Scores : Tokens start with hard modality IDs (0/1 for vision/text) and are progressively refined into layer-wise soft scores M\in[0,1] . We estimate them via (i) attention-accumulated (depicted here) or (ii) Gaussian-statistics estimation (detailed in Fig. 4 ). Expert Binning : experts learn bin-level modality preference for deployment efficiency. Mutual Information Regularization: maximizes mutual information between soft modality scores ( M ) and bins ( B ), inducing modality preference. 2.1 Mixture-of-Experts Sparsely-gated MoE architectures enable conditional computation through sparse activation and efficient gating 45 ; 14 ; 27 . Recent works like DeepSeek-V3 further improve routing efficiency, load balancing, and communication performance 43 ; 68 ; 34 . Mutual-information (MI) constraints have been explored to regulate expert selection. For expert-task alignment, Mod-Squad and MTHL maximize MI between tasks and experts 8 ; 7 . For expert-token alignment, ModuleFormer and CoMoE maximize MI between tokens and modules through entropy balancing or contrastive learning 48 ; 15 . However, these methods overlook modality differentiation in multimodal models. Recent VLMs increasingly adopt MoE backbones 60 ; 51 ; 22 ; 19 ; 55 ; 52 . Related approaches also include LoRA-based expert expansion 59 ; 6 ; 46 and converting dense VLMs to