FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation
Srijith Nair, Michael Lin, Amirreza Talebi, Peizhong Ju, Elizabeth Bentley, Jia Liu · 2025
Read FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation by Srijith Nair, Michael Lin, Amirreza Talebi, Peizhong Ju, Elizabeth Bentley, Jia Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23182.txt.
Introduction
Srijith Nair and co-authors speed up a privacy-preserving way to train AI models across many devices, in a 2025 paper. In split learning, each phone or hospital trains only the first slice of a model and sends intermediate results to a server, keeping raw data local. The catch is sequence: clients wait on the server round by round, so training crawls.
Their method, FSL-SAGE, gives each client a small helper model that imitates the server's feedback. Clients can then train in parallel with estimated gradients, syncing with the real server only periodically. The authors prove a convergence rate matching the classic federated averaging method, while cutting communication and client memory. Experiments beat earlier split-learning variants on both cost and accuracy.
Catalog record
| CREATOR | Srijith Nair, Michael Lin, Amirreza Talebi, Peizhong Ju, Elizabeth Bentley, Jia Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation Srijith Nair, Michael Lin, Amirreza Talebi, Peizhong Ju, Elizabeth Bentley, Jia Liu Abstract Collaborative training methods like Federated Learning (FL) and Split Learning (SL) enable distributed machine learning without sharing raw data. However, FL assumes clients can train entire models, which is infeasible for large-scale models. In contrast, while SL alleviates the client memory constraint in FL by offloading most training to the server, it increases network latency due to its sequential nature. Other methods address the conundrum by using local loss functions for parallel client-side training to improve efficiency, but they lack server feedback and potentially suffer poor accuracy. We propose FSL-SAGE ( F ederated S plit L earning via S mashed A ctivation G radient E stimation), a new federated split learning algorithm that estimates server-side gradient feedback via auxiliary models. These auxiliary models periodically adapt to emulate server behavior on local datasets. We show that FSL-SAGE achieves a convergence rate of 𝒪 ( 1 / T ) \mathcal{O}(1/\sqrt{T}) , where T T is the number of communication rounds. This result matches FedAvg, while significantly reducing communication costs and client memory requirements. Our empirical results also verify that it outperforms existing state-of-the-art FSL methods, offering both communication efficiency and accuracy. 1 Introduction 1) Background and Motivation: In the recent years, large foundation models ( Devlin et al. 2019 ; Radford et al. 2021 ; Ramesh et al. 2022 ) and LLMs ( OpenAI et al. 2024 ; Touvron et al. 2023 ) pretrained on large corpora of data have demonstrated near human-level performance on a variety of tasks. These models demand heavy computational resources and very large datasets for training ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . The datasets are often owned by different organizations, making it impossible for a single entity to train a model on all the data ( Sani et al. 2024b ) . Distributed learning paradigms like federated learning (FL) ( McMahan et al. 2016 ) and split learning (SL) ( Vepakomma et al. 2018 ) , which preserve data privacy while leveraging the compute power of various clients, have become increasingly popular in LLM pretraining ( Sani et al. 2024a ) and fine-tuning ( Sun et al. 2024 ; Wang et al. 2024 ) . Despite their benefits, FL and SL both have some limitations. Although FL can preserve data privacy by training the model on the client and aggregating the client-side models at a server, an inherent assumption of FL is that client devices have enough computational power to process the model, which is impractical for large models. SL, on the other hand, reduces compute requirements for clients by splitting the model into two parts, processing a smaller piece at the client, and offloading the other piece to the server. This, however, implies that processing the complete model in each iteration involves transmitting the cut-layer features and gradients back and forth between the client and the server. Moreover, SL loses the benefits of data parallelism in FL, since the server needs to assist gradient computation for each client in a sequential fashion. Consequently, SL suffers significant increases in communication load, power, and latencies. To reduce the communication overheads incurred by SL, some recent works ( Han et al. 2021 ; Mu & Shen 2023 ) propose to use auxiliary models at the clients. These auxiliary models continue the forward propagation of the client-side model to compute a local loss function, which is then used to update both models via gradient descent. This eliminates the need for awaiting the server’s response, which significantly speeds up the training process. Meanwhile, the server-side model is updated at a lower frequency by using the cut-layer features from the clients. While enjoying the benefit that the clients can conduct training in parallel, this can significantly reduce communication costs and latencies. However, the client-side models do not get any feedback from the server-side model, i.e., the clients are only updated using the auxiliary models and thus never get trained via the server-side model. To overcome these limitations, we propose a new approach called F ederated S plit L earning via S mashed A ctivation G radient E stimation (FSL-SAGE). Note that the auxiliary model in other FSL frameworks can be interpreted as mimicking the server-side model at each client. Since, from the client-side model’s perspective, the role of the server-side model is to provide the gradients of the loss function with respect to the cut-layer features, we train the auxiliary models to estimate these cut-layer gradients returned by the server-side model, which are then used to update the client-side models. This allows FSL-SAGE to continue to enjoy the benefits of FL and SL, while using the auxiliary models to accurately play the server’s role. Also, we propose to update the auxiliary models periodically in a less frequent fashion, thus significantly reducing communication costs. 2) Technical Challenges: However, due to the following technical challenges, establishing finite-time convergence rate performance guarantee for FSL-SAGE is highly non-trivial and necessitates new proof and analysis techniques. First, as will be shown later in Fig. 1 , there are multiple coupled time-scales involved during training, which include 1) the local step, whereby the client-side model is updated over a mini-batch of local data; 2) the lesser frequent server step, where the server-side model is updated given the cut-layer features; 3) a federated averaging step for the client-side models; and finally, 4) the least frequent alignment step, where the auxiliary models are updated to match the server-side model. Also, one needs to account for the auxiliary alignment step, which has not been studied in conventional SL methods (e.g., SplitFed ( Thapa et al. 2022 ) or in the precursor to this work ( Mu & Shen 2023 ) ). Exacerbating the problems in FSL-SAGE convergence rate analysis is the fact that the auxiliary models inject highly heterogeneous estimation errors during the training process, which are further coupled with all four time-scales. 3) Main Contributions: The key contribution of this paper is that we overcome the aforementioned technical challenges and rigorously establish the finite-time convergence rate guarantees of our proposed FSL-SAGE. Our main technical results are summarized as follows: • We propose a new federated split learning algorithm called FSL-SAGE to train models that are too large to be trained on commodity client devices. By using a periodically aligned auxiliary model at each client to estimate the server-side model response, FSL-SAGE enables large model federated training, while enjoying low communication costs and the same data parallelism as in conventional FL. • We establish a general finite-time stationarity convergence rate bound for FSL-SAGE. Based on this general result, we further show that under a mild in-expectation PAC-learnability assumption on the auxiliary models, FSL-SAGE can achieve an \mathcal{O}(1/\sqrt{T}) convergence rate, which is the same as those of state-of-the-art FL algorithms even with heterogeneous datasets. Our analysis sheds theoretical lights on the impacts of the class of auxiliary model functions on the final-time of convergence, which could be of independent interests to other FSL methods with auxiliary models. • We further propose a “lazy version” of our FSL-SAGE method, where the auxiliary models