Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing
Song Gao ∗ , Shusen Jing, Shuai Zhang † , Yue Wang ‡ , Xiangwei Zhou § , Songyang Zhang ∗ · 2025
Read Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing by Song Gao ∗ , Shusen Jing, Shuai Zhang † , Yue Wang ‡ , Xiangwei Zhou § , Songyang Zhang ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01743.txt.
Introduction
Song Gao and six colleagues move mixture-of-experts models, the architecture behind many large AI systems, onto mobile edge networks. In their Networked Mixture-of-Experts system, no single device holds the whole model. Clients answer inference requests collaboratively. They route each task to neighbors whose expertise fits, then aggregate the replies.
Training such a scattered system is the hard part, since edge devices are short on storage, compute, and shareable data. The authors propose a federated learning framework that mixes supervised and self-supervised learning to balance personalization with generalization while keeping communication efficient and data private. Extensive experiments demonstrate the system's effectiveness. They also give practical guidance on how the training algorithms behave. This is a step toward running large-model inference across the devices people actually carry.
Catalog record
| CREATOR | Song Gao ∗ , Shusen Jing, Shuai Zhang † , Yue Wang ‡ , Xiangwei Zhou § , Songyang Zhang ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01743
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing Song Gao ∗ , Shusen Jing, Shuai Zhang † , Yue Wang ‡ , Xiangwei Zhou § , Songyang Zhang ∗ Abstract Recent advancements in large artificial intelligence models (LAMs) are driving significant innovations in mobile edge computing within next-generation wireless networks. However, the substantial demands for computational resources and large-scale training data required to train LAMs conflict with the limited storage and computational capacity of edge devices, posing significant challenges to training and deploying LAMs at the edge. In this work, we introduce the Networked Mixture-of-Experts (NMoE) system, in which clients infer collaboratively by distributing tasks to suitable neighbors based on their expertise and aggregate the returned results. For training the NMoE, we propose a federated learning framework that integrates both supervised and self-supervised learning to balance personalization and generalization, while preserving communication efficiency and data privacy. We conduct extensive experiments to demonstrate the efficacy of the proposed NMoE system, providing insights and benchmarks for the NMoE training algorithms. I Introduction The recent wave of progress in large artificial intelligence models (LAMs) has triggered a variety of novel technologies, such as large language models (LLMs), vision-language models (VLMs), and artificial intelligence (AI) agents [ 1 ] , which present exciting opportunities for next-generation wireless communications. Leveraging powerful model architectures, a massive number of parameters, and abundant training data, LAM has achieved great successes in supporting large-scale wireless services, such as intelligent beam, channel state information (CSI) feedback, and semantic communications [ 2 , 3 ] . However, with the expansion of the Internet of Things (IoTs) and mobile devices, the training and deployment of LAM in edge computing have emerged as critical challenges, primarily due to the limited local computational capacity and distributed data storage. There is an urgent need for an efficient distributed LAM system to empower next-generation mobile computing. As large foundation models continue to evolve, the Mixture-of-Experts (MoE) paradigm has emerged as a novel learning framework that enables scaling LAMs to enormous sizes while maintaining manageable computational costs. This is achieved by dividing the model into a set of smaller, specialized “expert” subnetworks, of which only a subset is activated during inference [ 4 ] . With its sparsely activated architecture and expert specialization, MoE can significantly reduce computational costs while maintaining strong performance on heterogeneous data, making it a promising backbone for distributed LAM systems. Existing works on MoE-related distributed learning focus on integrating MoE with federated learning (FL) to mitigate the heterogeneous data issue. Typical examples include Fedmoe [ 5 ] , pFedMoE [ 6 ] , and PFL-MoE [ 7 ] , which aim to enhance the personalized federated learning performance; however, they ignore the constraints of computation capacity for each client. To this end, Chen et al. introduce a resource-aware [ 8 ] federated MoE, where load balance and local computational cost are jointly optimized. In addition, Qin et al. propose an optimal expert selection for distributed MoEs on the wireless edge, in which an optimization-based approach is introduced to address resource constraints [ 9 ] . Other MoE-based edge computing works also include channel-aware MoE [ 10 ] and HetuMoE [ 11 ] . Fig. 1: An overview of centralized MoE (left) and NMoE (right). The traditional MoE requires all components in the same edge device. In the proposed NMoE, each client locally deploys a cross-shared feature extractor, a cross-shared gating network, and a personalized expert. During inference, a client first processes its input data through FE to obtain latent features. These features are then passed to the gating network, which determines the most suitable experts (either local or from neighboring clients) for handling the data. The client subsequently distributes the extracted features according to the gating decisions, enabling selected experts to perform specialized inference. Despite some successes, these works assume that a complete MoE structure can be deployed on each client, overlooking the fact that limited local computational resources cannot support all expert networks simultaneously when they are activated during training. Therefore, instead of conducting federated training on the entire MoE-based LAM model, a more practical approach is to partition the MoE network into smaller components and deploy them across different edge devices. This allows the system to leverage computational resources from a larger number of edge nodes through efficient coordination over communication networks. However, such a setup has not yet been explored in the context of edge computing. The associated issues regarding data heterogeneity, privacy preservation, communication efficiency, model personalization, and generalization remain open challenges. In this work, we introduce networked mixture-of-expert (NMoE) as the first framework for the split and distributed MoE over mobile edges in communication networks. In the proposed NMoE system, each client locally deploys three subnetworks: a cross-shared feature extractor, a cross-shared gating network, and a personalized expert. During inference, a client first processes its input data through the feature extractor (FE) to obtain latent representations. Next, these features are passed to the gating network, which determines the most suitable experts (either local or from neighboring clients) to handle the data. The client subsequently distributes the extracted features according to the gating decisions, allowing selected experts to perform specialized inference. Finally, the outputs from the activated experts are aggregated and returned to the originating client. In addition, we propose several training strategies for the FE, gating network, and experts. For the FE, we design two federated training approaches that integrate supervised and self-supervised losses, respectively, to enhance both global representation quality and generalization. For the gating network, we introduce three variants, namely random gating (RanGate), rolling gating (RollGate), and federated gating (FedGate), which prioritize low communication and computation costs, privacy preservation, and model performance, respectively. Finally, each local expert is trained using a frozen FE on local datasets to preserve personalization and adapt to client-specific data distributions. To the best of our knowledge, we are the first to introduce and investigate NMoE. We make extensive experiments on these strategies, providing benchmarks and insights for training NMoE. II System Description In this section, we introduce the overall setup of NMoE. II-A Preliminary of MoE We first introduce the conventional centralized mixture-of-expert [ 4 ] . As shown in the left part of Fig. 1 , a classic MoE consists of a feature extractor F(\cdot) for latent embedding, a gating network G(\cdot) for expert selection, a set of n experts H_{i}(\cdot) for task specialization, and an aggregation module for combining outputs from selected experts. Given a data sample x , a latent h_{F}=F(x) is extracted from the feature extractor. The gating network G(\cdot) determines the contribution of each expert i denoted by G(h_{F})_{i} . Designing the aggregator as a weighted sum, the final output of the MoE is calculated by y^{\prime}=\sum_{i=1}^{n}G(h_{F})_{i}H_{i}(h_{F}). (1) Usually, Top- k expert selection is applied for the gating network, where G(h_{F})=Softmax(Topk(h_{F}\cdot W_{G}+\epsilon,k)) . Here, W_{G} is the weight of the gate, and \epsilon is a tiny noise for perturbation to improve the load balance. k represents the number of selected experts. Let v be the input of Topk , and v_{i} represent the i- th element of the input v . Then, we have Topk(v,k)_{i}=\begin{cases}v_{i},&\text{if }v_{i}\in\text{Top-}k\text{ values}\\ -\infty,&\text{otherwise}\end{cases}. This type of gate