FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE

Khiem Le, Tuan Tran, Ting Hua, Nitesh V. Chawla · 2025

Read FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE by Khiem Le, Tuan Tran, Ting Hua, Nitesh V. Chawla (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16600.txt.

Introduction

Khiem Le, Tuan Tran, Ting Hua, and Nitesh Chawla propose a 2025 way to fine-tune large language models across many sites. In federated learning, each site trains on its own data and shares updates only. Sites differ in computing power, so past methods shrank shared parts to fit weak sites. That shrink step lost key detail and hurt results.

Their method, FLAME, keeps the full shared parts intact. Fit comes instead from how many expert units each site turns on. Two fixes handle snags from this design. A light rescaling step evens out output size when only some experts run. A second step weights updates by which experts each site trained. Tests across varied compute settings show FLAME ahead of past methods. The authors offer it as a robust choice for mixed fleets.

Catalog record

CREATORKhiem Le, Tuan Tran, Ting Hua, Nitesh V. Chawla
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16600

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE Khiem Le, Tuan Tran, Ting Hua, Nitesh V. Chawla Abstract Existing resource-adaptive LoRA federated fine-tuning methods enable clients to fine-tune models using compressed versions of global LoRA matrices, in order to accommodate various compute resources across clients. This compression requirement will lead to suboptimal performance due to information loss. To address this, we propose FLAME, a novel federated learning framework based on the Sparse Mixture-of-Experts (SMoE) architecture. Unlike prior approaches, FLAME retains full (uncompressed) global LoRA matrices and achieves client-side adaptability by varying the number of activated experts per client. However, incorporating SMoE into federated learning introduces unique challenges—specifically, the mismatch in output magnitude from partial expert activation and the imbalance in expert training quality across clients. FLAME tackles these challenges through a lightweight rescaling mechanism and an activation-aware aggregation scheme. Empirical results across diverse computational settings demonstrate that FLAME consistently outperforms existing methods, providing a robust and effective solution for resource-adaptive federated learning. 1 Introduction Large language models (LLMs) have achieved remarkable success across a wide range of tasks, largely due to extensive pre-training on massive datasets achiam2023gpt ; dong2023towards ; kelly2023bing ; thirunavukarasu2023large . While pre-trained LLMs demonstrate remarkable general abilities, fine-tuning remains essential for adapting these models to specific domains and applications anisuzzaman2025fine ; parthasarathy2024ultimate . However, centralizing sensitive data for fine-tuning raises significant privacy concerns voigt2017eu ; pardau2018california making federated learning (FL) an increasingly important paradigm that enables model improvement while preserving data locality zhang-etal-2023-fedpet ; kuang2024federatedscope ; woisetschlager2024federated ; wu2025survey . Existing approaches cho-etal-2024-heterogeneous ; bai2024federated typically maintain global LoRA matrices on central servers, while requiring each client to use a compressed version of the global LoRA matrices that aligns with its own computational constraints—typically by reducing the rank. These methods balance heterogeneous client capabilities primarily through matrix decomposition, particularly Singular Value Decomposition (SVD), operating under the assumption that higher-rank matrices preserve more information. Such a design deals with heterogeneity by allowing resource-constrained clients to use lower-rank approximations of the global LoRA matrices, while resource-rich clients can utilize higher-rank representations. Research has demonstrated that the importance ranking of singular values from SVD does not always align optimally with preserving LLM performance on downstream tasks hsulanguage ; hua2022numerical . Furthermore, this matrix decomposition-based strategy has inherent limitations: in order to accommodate diverse client capabilities, all local models are inherently forced to discard part of the knowledge encoded in the global LoRA matrices, which is obviously suboptimal. Besides, our thorough investigation reveals crucial limitations of these methods. A deeper examination of FLOPs in our evaluation reveals that fine-tuning with LoRA matrices of smaller ranks does not remarkably reduce computational loads, since the computational demands for the base forward pass remain unchanged. This indicates that existing methods fundamentally fail to enable clients to complete fine-tuning using computational loads truly tailored to their resource budgets, representing a misleading direction in resource-adaptive federated learning. To address these limitations, we revisit the Sparse Mixture-of-Experts (SMoE) architecture and propose FLAME ( F ederated L earning with A daptive sparse M ixture-of- E xperts), a novel federated learning framework that leverages SMoE to enable genuine resource-adaptive fine-tuning. As illustrated in Figure 1 , FLAME allows each client to fine-tune using full (uncompressed) global LoRA matrices while varying the number of activated experts based on available compute capacity. Importantly, our examination of FLOPs verifies that this approach remarkably reduces computational loads since the base forward pass costs are accordingly reduced. In contrast to existing methods, FLAME truly enables clients to complete fine-tuning using computational loads tailored to their resource budgets. Furthermore, FLAME’s approach offers significant advantages during deployment. By fine-tuning the model with global LoRA matrices while using smaller numbers of activated experts in SMoE layers, FLAME facilitates deploying the model with reduced expert activation during inference, thereby significantly enhancing deployment efficiency. Figure 1 : An illustration of resource-adaptive federated fine-tuning with FLAME. However, this design introduces two key challenges. First, partial expert activation creates output magnitude mismatches compared to full-capacity execution. FLAME addresses this through a lightweight learnable rescaling mechanism that adaptively calibrates outputs across different activation patterns. Second, an imbalance in expert activation frequency across clients results in an imbalance in the quality of their trained LoRA matrices, which can distort the global model after aggregation. To solve this, FLAME incorporates an activation-aware federated averaging scheme that incorporates activation frequency when generating global LoRA matrices, ensuring proper weighting of client contributions for each expert. Our contributions can be summarized as follows: • Limitation analysis of existing methods: We thoroughly investigate resource-adaptive federated fine-tuning LLMs and identify crucial limitations of existing methods, particularly their failure to enable true computational load adaptation tailored to client resource budgets. • A novel adaptive SMoE framework for federated learning: We introduce FLAME, a novel federated learning framework that leverages sparse mixture-of-experts (SMoE) architecture to enable resource-adaptive fine-tuning without compromising the expressive power of global LoRA matrices. Unlike existing compression-based approaches, our method maintains full global LoRA matrices while varying the number of activated experts according to client computational capabilities. • Activation-aware aggregation scheme: We develop an activation-aware federated averaging scheme that incorporates expert activation frequency across clients to generate balanced global LoRA matrices, addressing the shortcomings of standard federated averaging. • Comprehensive performance evaluation: We demonstrate through extensive experiments on instruction-following tasks that FLAME achieves significantly better performance than existing methods across various computational settings and data distributions. 2 Methodology 2.1 Preliminaries We study fine-tuning large language models using Low-Rank Adaptation (LoRA) in a federated setting where data remains distributed across clients due to privacy concerns. A key challenge in this environment is accommodating the heterogeneous computational capabilities of participating devices through resource-adaptive federated fine-tuning. Existing federated LoRA fine-tuning methods utilize a central server to coordinate training across clients. Each client holds a local dataset D_{i} and operates under its resource constraint \beta_{i} . The server initializes and distributes global LoRA matrices of rank r , \text{A}\in\mathbb{R}^{m\times r} and \text{B}\in\mathbb{R}^{r\times n} , for fine-tuning the frozen base model \text{W}\in\mathbb{R}^{m\times n} . Each client fine-tunes the LoRA matrices on its local data: \text{A}_{i}=\text{A},\quad\text{B}_{i}=\text{B}, (1) \min\frac{1}{|D_{i}|}\sum_{x\in D_{i}}\ell(\text{W},x\mid\beta_{i})\quad\text{s.t.}\quad h=\text{W}x+\text{A}_{i}\text{B}_{i}x. (2) After local updates, clients return \text{A}_{i} and \text{B}_{i} to the server. Global LoRA matrices are then updated via federated averaging mcmahan2017communication ; li2020on . Each client’s contribution is weighted by its dataset size \gamma_{i}=|D_{i}| , reflecting the training intensity of its local updates: \gamma_{i}=|D_{i}|,\quad\text{for }1\leq i\leq N, (3) \text{A}=\frac{\sum_{i=1}^{N}\gamma_{i}\text{A}_{i}}{\sum_{i=1}^{N}\gamma_{i}},\quad\text{B}=\frac{\sum_{i=1}^{N}\gamma_{i}\text{B}_{i}}{\sum_{i=1}^{N}\gamma_{i}}. (4) This framework ensures stable global aggregation by diminishing the impact of low-quality updates from clients with less data. (a) AlpaGasus ( \alpha = 5) (b) AlpaGasus ( \alpha = 0.5) (c) Dolly ( \alpha = 5) (d) Dolly ( \alpha = 0.5) Figure 2 : A highlight of the activation frequency of experts across clients in our experiments. The heatmaps display activation frequencies of all 64 experts (x-axis) across 4 clients (y-axis) for both AlpaGasus and Dolly datasets under different data heterogeneity settings. 2.2 FLAME: Federated Learning with Adaptive MoE Our proposed FLAME considers an SMoE-based model. We denote M matrices of parameters in SMoE layers as \{\text{W}^{j}\in\mathbb{R}^{m\times n}\}_{j=1}^{M} , corresponding to M experts managed by a router that dynamically selects k<M activated experts for processing input tokens based on

Read the complete text