Efficient Training of Large-Scale AI Models through Federated Mixture-of-Experts: A System-Level Approach

Xiaobing Chen, Boyang Zhang, Xiangwei Zhou, Mingxuan Sun, Shuai Zhang, Songyang Zhang, and Geoffrey Ye Li · 2025

Read Efficient Training of Large-Scale AI Models through Federated Mixture-of-Experts: A System-Level Approach by Xiaobing Chen, Boyang Zhang, Xiangwei Zhou, Mingxuan Sun, Shuai Zhang, Songyang Zhang, and Geoffrey Ye Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05685.txt.

Introduction

Xiaobing Chen and co-authors sketch a shop-floor plan to train huge AI models across split sites, with no raw data pooled. Two tricks fuse: shared learns that keep data home, and expert blends that route each task to a sub-net. The blend is strong but hard to run when sites hold unlike gear and experts crowd some sites. Their 2025 piece names the gap: no firm way to match clients to experts on the fly.

Their sketch scores each client's fit in real time, tracks expert load round the web, and files each client's strength. Match calls bend as loads shift. Fair loads can cut chat rounds pre the model settles. The plan is a frame, not a built tool, and aims at scale with grit.

Catalog record

CREATORXiaobing Chen, Boyang Zhang, Xiangwei Zhou, Mingxuan Sun, Shuai Zhang, Songyang Zhang, and Geoffrey Ye Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05685

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Training of Large-Scale AI Models through Federated Mixture-of-Experts: A System-Level Approach Xiaobing Chen, Boyang Zhang, Xiangwei Zhou, Mingxuan Sun, Shuai Zhang, Songyang Zhang, and Geoffrey Ye Li Abstract The integration of Federated Learning (FL) and Mixture-of-Experts (MoE) presents a compelling pathway for training more powerful, large-scale artificial intelligence models (LAMs) on decentralized data while preserving privacy. However, efficient federated training of these complex MoE-structured LAMs is hindered by significant system-level challenges, particularly in managing the interplay between heterogeneous client resources and the sophisticated coordination required for numerous specialized experts. This article highlights a critical, yet underexplored concept: the absence of robust quantitative strategies for dynamic client-expert alignment that holistically considers varying client capacities and the imperative for system-wise load balancing. Specifically, we propose a conceptual system design for intelligent client-expert alignment that incorporates dynamic fitness scoring, global expert load monitoring, and client capacity profiling. By tackling these systemic issues, we can unlock more scalable, efficient, and robust training mechanisms with fewer communication rounds for convergence, paving the way for the widespread deployment of large-scale federated MoE-structured LAMs in edge computing with ultra-high communication efficiency. I Introduction The relentless advancement of large-scale artificial intelligence models (LAMs) has triggered numerous emerging technologies, such as large language models and vision foundation models, which offer unprecedented capabilities across a multitude of applications from scientific analysis to practical problem solving, including biomedical image processing, remote sensing and wireless communications [ 1 ] . Despite offering superior performance and transformative capabilities, these LAMs, with billions of parameters, demand immense computational resources and memory for both training and inference, limiting their deployment in practical scenarios, particularly in edge computing systems with a constrained computational capacity. Furthermore, the vast data for training LAMs are often geometrically distributed across various organizations or user devices, resulting in growing privacy concerns and logistical challenges related to data aggregation, which do not lend themselves to conventional centralized learning. Although distributed learning provides potential solutions, significant communication and computation overheads are introduced, particularly with streaming data and heterogeneous hardware capabilities [ 2 ] . To enhance the efficiency and privacy preservation for training LAMs in distributed systems, two powerful paradigms have recently gained prominence: Federated Learning (FL) and Mixture-of-Experts (MoE) [ 3 ] . By enabling collaborative model training without sharing raw data, FL offers a compelling path to address data privacy and learning efficiency in distributed edge computing [ 4 ] . On the other hand, MoE architectures provide an effective strategy to scale LAMs to enormous sizes while maintaining computational tractability by dividing the LAM into numerous smaller, specialized “expert” sub-networks, as shown in Fig. 1 , where only a subset of experts is activated during inference. Through sparsely activating optimal experts, MoE-structured LAM can significantly reduce the computation cost and enhance task performance. Thus, integration of FL and MoE can be a natural solution to enable efficient and effective LAMs in edge computing while preserving privacy. However, as an emerging technology, the federated training of MoE-structured LAM remains an ill-posed and underexplored problem. In this work, we will introduce the key concepts of FL for MoE-structured LAM and propose a conceptual system design. I-A Scope and Overview This work focuses on the federated training of MoE-structured LAM for edge computing. As shown in Fig. 1 , the distributed MoE-structured LAM system consists of a central server for global management and several clients (e.g., edge devices or deployed sensors) with different computational capacities and corresponding local data. Through FL, these clients aim to collaboratively refine a global MoE-structured LAM without sharing raw data. Due to the limitation of computational resources, each client can only be assigned to train a specific subset of experts simultaneously. After federated training, the most suitable experts are assigned to each client during inference. This synergistic design allows for efficient training and effective deployment of more powerful and specialized MoE-structured LAMs for real-world problems across diverse domains while preserving data privacy. Figure 1 : Proposed system of FL-enabled, MoE-structured LAM for edge computing: 1) Centralized MoE consists of a gating network and several expert networks, where the gating function routes each data sample to its most suitable expert for processing; 2) FL aims to collaboratively train the objective neural network by sharing gradients/models instead of raw data, which is capable of preserving data privacy; 3) Example of distributed MoE with FL: In a smart city sensor system, different edge devices such as vehicles, drones, and intelligent traffic infrastructure act as clients. Each client can be tasked with training a specific subset of experts (e.g., E1, E3, E5 for the traffic sensor in the figure) from a large, comprehensive MoE model based on its unique local data. Through FL, these clients collaboratively refine a global MoE LAM capable of complex tasks, such as intelligent traffic management or public safety monitoring. Despite the intuition of this distributed MoE-structured LAM system design, realizing its efficient and scalable training in a federated manner remains an underexplored problem. Resulting from the demand for specialization of each expert and heterogeneous data distribution, the first challenge lies in the personalized training of experts for the appropriate clients. Moreover, the varying occupancy of limited local computational resources across different clients, together with the dynamic communication channel condition, further increases the difficulty of client-expert alignment, motivating the development of resource-aware strategies for MoE training. Although some existing studies have discussed the deployment and gating of MoE-structured LAM in edge computing, considering channel quality [ 5 ] or end-to-end latency [ 6 ] , they only focus on training in a centralized setup, while the practicalities of federated training for MoE remain challenging. I-B Contents and Structures This article aims to delineate the critical area of FL-enabled MoE-structured LAMs for edge computing. Specifically, we highlight the current lack of quantitative and holistic analysis and optimization strategies to address the intricate interplay among three key system-level factors: 1) the heterogeneous system capacities of clients; 2) the dynamic client-expert alignment to match appropriate clients to each expert; and 3) the overarching system-wise load balancing for efficient and effective MoE training. To address the aforementioned challenges, we propose a conceptual system design for dynamic client-expert alignment considering both varying client capacities and system-wise load balancing within the FL setup. Experimental results validate the promising potential for the proposed system design. The remainder of this work is organized as follows. We first review the foundations of FL and MoE, together with the current stage of integrating FL and MoE in Section II . Then, we illustrate the major challenges of FL-enabled MoE from the systematic perspective and propose our conceptual design in Section III . Following the discussion of additional challenges and future opportunities in Section IV , we summarize our work in Section V . II Federated Learning and Mixture-of-Experts Before stepping into the FL-enabled, MoE-structured LAM design for edge computing, we first provide an overview of the current stage of FL and MoE, together with their integration. II-A Federated Learning FL is a decentralized learning paradigm in which several clients collaboratively train a shared LAM without exchanging their raw local data, coordinated by a central server [ 4 ] . In a standard FL process, clients compute the

Read the complete text