Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
Fahao Chen, Jie Wan, Peng Li, Zhou Su, Dongxiao Yu · 2025
Read Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices by Fahao Chen, Jie Wan, Peng Li, Zhou Su, Dongxiao Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19078.txt.
Introduction
Fahao Chen and co-authors fine-tune giant sparse models across weak devices in a 2025 paper. Mixture-of-experts language models activate only a few expert parts per token. Even so, tuning them usually demands data-center hardware, not consumer cards. Privacy rules also bar pooling everyone's raw data in one place.
Federated tuning spreads work across many owners' machines without pooling raw data. Prior tries leaned on pruning or offloading with shaky assumptions about those systems. Their system, FLUX, profiles expert use cheaply through quantized local runs. It then schedules work to cut time-to-accuracy on modest home and office GPUs. MoE-specific smarts, not generic tricks, drive the speedup.
Catalog record
| CREATOR | Fahao Chen, Jie Wan, Peng Li, Zhou Su, Dongxiao Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19078
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices Fahao Chen, Jie Wan, Peng Li, Zhou Su, Dongxiao Yu Abstract . Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of participants. Existing working attempts to fill this gap through model quantization, computation offloading, or expert pruning. However, they cannot achieve desired performance due to impractical system assumptions and a lack of consideration for MoE-specific characteristics. In this paper, we propose Flux , a system designed to enable federated fine-tuning of MoE-based LLMs across participants with constrained computing resources (e.g., consumer-grade GPUs), aiming to minimize time-to-accuracy. Flux introduces three key innovations: (1) quantization-based local profiling to estimate expert activation with minimal overhead, (2) adaptive layer-aware expert merging to reduce resource consumption while preserving accuracy, and (3) dynamic expert role assignment using an exploration-exploitation strategy to balance tuning and non-tuning experts. Extensive experiments on LLaMA-MoE and DeepSeek-MoE with multiple benchmark datasets demonstrate that Flux significantly outperforms existing methods, achieving up to 4.75 × \times speedup in time-to-accuracy. 1. Introduction Sparsely activated large language models (LLMs), particularly those leveraging the Mixture-of-Experts (MoE) architecture, have recently demonstrated strong capability in various NLP tasks ( Lepikhin et al. 2021 ; Riquelme et al. 2021 ; Fedus et al. 2022 ) . An MoE-based LLM consists of multiple specialized expert networks, each tailored to handle specific types of input data. Before deployment, it typically undergoes a fine-tuning process, where it is re-trained on domain-specific data to enhance its adaptability and accuracy. In many real-world scenarios, such data are distributed across multiple participants (e.g., organizations or companies) that are either unwilling or legally restricted from sharing their data. This setting naturally aligns with the principles of federated learning ( McMahan et al. 2017 ) , which enables collaborative model training by exchanging model updates instead of raw data, thereby preserving data privacy. Despite extensive research in federated learning, an efficient solution for federated fine-tuning of MoE-based LLMs remains elusive. The primary challenge stems from the massive model size, which often exceeds the computing capabilities of individual participant. Therefore, traditional federated learning techniques, such as ( Lai et al. 2021 ; Li et al. 2022 ; Cai et al. 2023b ; Xu et al. 2024a ; Ching et al. 2024 ; Khan et al. 2024 ) , which rely on full-model training and parameter exchange, are not directly applicable. Recent studies ( Isaksson et al. 2022 ; Guo et al. 2021 ) have explored federated MoE fine-tuning by focusing on expert layers, demonstrating its effectiveness in certain scenarios. However, these methods primarily target small-scale MoE models and assume that the entire model can fit within local computing devices, limiting their scalability and applicability to LLM deployments. To adapt the MoE-based LLM to constrained computing resources, several optimization techniques can be employed. Quantization is a widely used method for model compression by reducing the precision of model parameters from high-bit floating point representations (e.g., FP32) to lower-bit formats (e.g., INT8). Previous studies have demonstrated that quantization is highly effective for MoE inference ( Frantar and Alistarh 2024 ; Yi et al. 2025 ) . However, fine-tuning a quantized MoE model often fails to achieve the desired performance, because of accumulated precision errors during fine-tuning. Another approach is to offload some experts to main memory (RAM) and dynamically transfer them to the GPU when needed ( Hwang et al. 2024 ; Kong et al. 2024 ; Cao et al. 2025 ) . However, frequent data transfers between main memory and the GPU introduce substantial latency, significantly slowing down fine-tuning. Some recent studies have explored a more radical approach by pruning “unimportant” experts to reduce the model’s computational and memory footprint ( Lu et al. 2024 ; Chowdhury et al. [n. d.] ; Mei et al. 2024 ) . However, our findings indicate that these seemingly “unimportant” experts still be crucial for model convergence. Removing them could slow down fine-tuning or even compromise overall performance. In this paper, we present Flux , a system designed to enable federated fine-tuning of MoE-based LLMs across participants with constrained computing resources (e.g., consumer-grade GPUs), aiming to minimize time-to-accuracy. The core idea of Flux is to let each participant construct a compact MoE model that closely approximates the full model in both tuning behavior and learned representations. Specifically, experts on each participant are categorized into tuning experts and non-tuning ones , according to their contributions to fine-tuning. Tuning experts are preserved in their original size and updated during tuning, while non-tuning experts are merged and frozen to reduce memory and computational overhead. Note that expert roles, tuning or non-turning, could dynamically change during the runtime. Although this idea is promising, there are several critical system-level challenges that must be addressed to ensure efficiency in practical settings, particularly given non-IID data and heterogeneous computing and communication resources across participants. Profiling expert activation . The first challenge Flux faces is efficiently profiling expert activation to evaluate their contributions to fine-tuning. Unlike traditional dense models, where all parameters are trained on the entire dataset, MoE models activate only a subset of experts for each input, meaning each expert is trained on a specific subset of tokens. By profiling expert activation, we can determine which data should be used to train tuning experts and assess the importance of non-tuning experts. Some existing works ( Zhang et al. 2024a ; Mei et al. 2024 ) have explored selecting a subset of highly active experts for training. However, they assume that profiling information is readily available. In federated settings, profiling on resource-constrained participants is highly challenging. Running the full MoE model to measure expert activation is computationally infeasible, and delegating this process to third parties compromises data privacy. Flux addresses this challenge by introducing a quantization-based local profiling method, leveraging a quantized MoE model to efficiently estimate expert activation frequency and determine the data processed by each expert. This approach significantly reduces computational overhead while ensuring a reliable assessment of expert relevance for fine-tuning. Merging non-tuning experts . Non-tuning experts must be efficiently merged to fit within resource constraints while preserving model accuracy. Existing studies ( He et al. 2023 ; Li et al. 2024b ) have demonstrated the benefits of expert merging, but they primarily focus on global model compression, merging experts based solely on activation frequency. However, this simplistic approach overlooks critical factors influencing fine-tuning effectiveness, leaving room for substantial improvement in minimizing its negative impact. Flux introduces an adaptive layer-aware merging strategy , where memory allocation for non-tuning experts is determined based on expert activation distributions and error accumulation across layers. This ensures that layers prone to higher error propagation retain greater expert diversity, mitigating performance degradation. Additionally, instead of relying solely on activation frequency, Flux incorporates token attention scores to assess expert importance during merging, ensuring that the process preserves critical model behaviors. Expert role assignment . The final challenge lies in expert role assignment, i.e., determining whether an expert should be tuning or non-tuning, under strict computational and memory constraints at participants. Increasing the number of tuning experts accelerates convergence but prolongs local training time. Moreover, they encroach on the memory space