Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October

YongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li, Zihao Wu, Chaohu Liu, Linli Xu · 2025

Read Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October by YongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li, Zihao Wu, Chaohu Liu, Linli Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16448.txt.

Introduction

Yongxiang Hua leads six co-authors in a 2025 study on routing inputs inside large image-and-text models. These models use expert parts, where each input goes to a few experts. Old routers pick experts by score match, which blurs who should own what. Their design, Input Domain Aware MoE, routes by input zone instead.

A chance-based mix model splits the input space into clear zones. Each expert then owns a zone and trains apart from the main task. That split keeps training stable and picks crisp, not vague. Tests on image-and-text tasks beat older expert models on score and balance. Balanced expert use means less waste and faster runs.

Catalog record

CREATORYongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li, Zihao Wu, Chaohu Liu, Linli Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16448

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755754 ISBN: 979-8-4007-2035-2/2025/10 5738 CCS: Computing methodologies Computer vision CCS: Computing methodologies Natural language processing CCS: Computing methodologies Ensemble methods YongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li, Zihao Wu, Chaohu Liu, Linli Xu Abstract . Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models, offering substantial capacity while maintaining computational efficiency through dynamic, sparse activation of experts. However, existing routing mechanisms, typically based on similarity scoring, struggle to effectively capture the underlying input structure. This limitation leads to a trade-off between expert specialization and balanced computation, hindering both scalability and performance. We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space. By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization. Unlike conventional approaches, our routing mechanism is trained independently of task-specific objectives, allowing for stable optimization and decisive expert assignments. Empirical results on vision-language tasks demonstrate that our method consistently outperforms existing sMoE approaches, achieving higher task performance and improved expert utilization balance. 1. Introduction Vision-language models (VLMs) have demonstrated remarkable capabilities, but their immense computational demands pose significant scaling challenges. To address this, sparsely-gated Mixture-of-Experts (sMoE) was introduced in Large Language Models (LLMs) ( Lepikhin et al. 2020 ) and has since established itself as a dominant solution, finding widespread applications across models of varying scales and domains ( Dai et al. 2024 ; Zhang et al. 2024 ; Yu et al. 2023a ; Wu et al. 2024b ; Gao et al. 2024 ; Cao et al. 2023 ; Liu et al. 2024f ) . At its core, sMoE employs conditional computation where each input token activates only a subset of specialized experts through a learned routing function, effectively reducing computational overhead while maintaining model capacity. However, the effectiveness of this approach ultimately hinges on the routing mechanism that assigns inputs to experts. (a) Comparative analysis of IDA-MoE against various routing mechanisms implemented in LLaVA MoE (b) Performance evaluation of IDA-MoE versus sota open-source models of comparable parameter size (2B) Figure 1 . Benchmark performance of IDA-MoE (ours) across visual question answering and multimodal reasoning tasks. The left panel ( Fig. 1(a) ) illustrates the effectiveness of different MoE routing implementations, while the right panel ( Fig. 1(b) ) demonstrates comparative performance against models of similar computational scale. Various routing methods have been proposed. Among them, the predominant approaches ( Lepikhin et al. 2020 ; Fedus et al. 2022 ; Dai et al. 2024 ; Liu et al. 2024b ; Liu et al. 2024d ) rely on similarity-based scoring, where routing probabilities are computed through softmax over token-expert alignment scores. This mechanism, trained end-to-end with the task objective, tends to develop a strong bias towards routing tokens to high-performing experts, creating a feedback loop that exacerbates load imbalance. To mitigate this, conventional approaches introduce auxiliary load balancing objectives ( Shazeer et al. 2017 ; Lepikhin et al. 2020 ; Zoph et al. 2022 ) that push per-token routing distributions toward uniformity across experts. However, this creates a fundamental contradiction: while the core principle of sMoE emphasizes specialized experts through conditional computation, load balancing forces indiscriminate token distribution, inevitably undermining expert specialization. This unresolved tension manifests in two critical problems: during training, routing instability leads to frequent changes in token-expert assignments, severely impacting sample efficiency ( Dai et al. 2022 ; Zoph et al. 2022 ) ; during inference, ambiguous routing decisions result in suboptimal expert selection, compromising model performance and robustness to distribution shifts ( Wu et al. 2024c ; Wang et al. 2024a ) . The root cause lies in coupling routing decisions with task optimization, which makes it contradictory to achieve balanced token allocation while preserving sharp, decisive routing. Specifically, when experts are evaluated solely on their contribution to the final loss, the system naturally concentrates computation on a small subset of versatile experts rather than developing complementary specializations(analysed in Section 3.2 ). This suggests a fundamental need to decouple routing from task-specific learning signals and instead ground it in the inherent structure of the input distribution. To this end, we propose Input Domain Aware MoE (IDA-MoE), which explicitly models the input space through a Gaussian Mixture Model (GMM) to create natural, data-driven routing boundaries. This probabilistic framework partitions inputs based on their distribution characteristics rather than loss contributions, where routing decisions are guided by the posterior probabilities rather than learned similarity scores, enabling experts to develop clear specializations while maintaining balanced utilization. By associating each expert with multiple mixture components (i.e., regions of the input distribution) rather than learning routing directly from task objectives, our approach creates a flexible mapping between input domains and expert specializations. This decoupling allows independent optimization of expert workload distribution (controlled by mixture priors P(k) ) while maintaining sharp, distribution-aware token routing - achieving load balance as an emergent property rather than an enforced constraint. Our contributions in this work are threefold. • First, we present a systematic analysis of existing MoE routing mechanisms, revealing the fundamental dilemma: the coupling between routing and task objectives inherently drives load imbalance, while conventional mitigation strategies often compromise training stability and prevent effective expert specialization. • Second, we introduce Input Domain Aware MoE (IDA-MoE), a novel architecture that explicitly models input distribution through probabilistic mixture modeling, effectively decoupling routing decisions from task optimization while maintaining precise expert specialization. • Third, through comprehensive experiments across multiple vision-language tasks, we demonstrate that IDA-MoE achieves superior performance compared to traditional routing approaches, exhibiting improved training stability, better expert utilization, and enhanced performance on VQA benchmarks. 2. Related Works Large Vision-Language Models. The emergence of powerful Large Language Models (LLMs) has enabled a new paradigm for multimodal learning, where frozen LLMs act as reasoning engines for vision-language tasks. Early approaches like BLIP-2 ( Li et al. 2023b ) and LLaVA ( Liu et al. 2023b ; Liu et al. 2024e ) proposed lightweight adapters (e.g., Q-Former, MLP projectors) to map visual features from pretrained encoders (e.g., CLIP ( Radford et al. 2021 ) ) into the LLM’s token space. This adapter-based paradigm has achieved substantial performance gains across diverse vision-language tasks. Recent works ( Young et al. 2024 ; Fu et al. 2024 ; Fu et al. 2025 ; Shen et al. 2025 ; Chen et al. 2024a ; Bai et al. 2025 ; Chen et al. 2024b ; Grattafiori et al. 2024 ) continue this paradigm by aggressively scaling both model parameters and training data. While this leads to improved capabilities, it also makes training and inference prohibitively expensive. To address these computational challenges, researchers have explored sparse Mixture of Experts (sMoE) architectures, which are adopted both in pre-training ( Bao et al. 2022 ; Bai et al. 2025 ; Wu et al. 2024a ) and via efficient upcycling of dense models ( Komatsuzaki et al. 2022 ; Lin

Read the complete text