FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li · 2024
Read FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence by Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13779.txt.
Introduction
Yichen Li and co-authors present a 2024 method for shared learning across devices that keeps learning new tasks over time. In such setups, each device holds its own data and no raw data leaves it. The hard part is learning new classes without forgetting old ones, a task most past methods solve by replaying saved old data.
Replay costs memory and can breach privacy, so the team turns to penalty methods that guard key weights instead. A known penalty method works well when all devices hold like data but fails when their data differ a lot. Their new method adapts that penalty for mixed data across devices. Tests beat top rivals, with less compute and no saved old data at all.
Catalog record
| CREATOR | Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13779
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li Abstract Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI , which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods. 1 Introduction Federated learning (FL) is to facilitate the collaborative training of a global deep learning model among multiple edge clients while ensuring the privacy of their locally stored data ( McMahan et al., 2017 ; Wang et al., 2023a ; Liu et al., 2024 ) . Recently, FL has garnered significant interest and found applications in diverse domains, including recommendation systems ( Yang et al., 2020 ; Li et al., 2024c ) and smart healthcare solutions ( Xu et al., 2021 ; Nguyen et al., 2022 ) . Typically, FL has been studied in a static setting, where the number of training samples does not change over time. However, in realistic FL applications, each client may continuously collect new data and train the local model with streaming tasks, leading to performance degradation on previous tasks ( Yang et al., 2024 ; Wang et al., 2024c ) . This phenomenon, known as catastrophic forgetting ( Ganin et al., 2016 ) , poses a significant challenge in the continual learning (CL) paradigm. This challenge is further compounded in FL settings, where the data on one client remains inaccessible to others, and clients are constrained by memory and computation resources. Table 1: Primary Directions of Progress in CFL. Analysis of the recent major techniques in the CFL system with the main contribution. Here, we focus on three common weak points about data rehearsal, computational overhead, and privacy concerns. Reference Contribution Common Weak Points Data Rehearsal Computational Overhead Privacy Concerns TARGET ( Zhang et al., 2023 ) Synthetic Exemplar ✓(Synthetic Sample) ✓(Distillation, Generator) ✕ FedCIL ( Qi et al., 2023 ) Generation & Alignment ✓(Synthetic Sample) ✓(ACGAN) ✓(Shared Generator) Re-Fed ( Li et al., 2024a ) Synergism ✓(Cached Sample) ✕ ✕ AF-FCL ( Wuerkaixi et al., 2023 ) Accurate Forgetting ✓(Synthetic Sample) ✓(NF Model) ✓(Shared Generator) SR-FDIL ( Li et al., 2024b ) Synergism ✓(Cached Sample) ✓(GAN Model) ✓(Shared Discriminator) GLFC ( Dong et al., 2022 ) Class-Aware Loss ✓(Cached Sample) ✕ ✓(Proxy Server) FOT ( Bakman et al., 2023 ) Orthogonality ✕ ✓(Multi-Heads,Projection) ✓(Task-ID) FedWeIT ( Yoon et al., 2021 ) Network Extension ✕ ✕ ✓(Task-ID) CFeD ( Ma et al., 2022b ) Two-sides KD ✓(Distillation Sample) ✓(Distillation) ✕ MFCL ( Babakniya et al., 2024 ) Data-Free KD ✓(Synthetic Sample) ✓(Extra Training) ✕ FedET ( Liu et al., 2023 ) Pre-training Backbone ✓(Cached Sample) ✓(Transformer) ✕ Ours Regularization ✕ ✕ ✕ To address this issue, researchers have studied continual federated learning (CFL), which enables each client to continuously learn from local streaming tasks. FedCIL is proposed in ( Qi et al., 2023 ) to learn a generative network and reconstruct previous samples for replay, improving the retention of previous information. The authors in ( Li et al., 2024a ) propose to selectively cache important samples from each task and train the local model with both cached samples and the samples from the current task. The authors in ( Dong et al., 2022 ; Dong et al., 2023 ) focus on the federated class-incremental learning scenario and train a global model by computing additional class-imbalance losses. The study in ( Ma et al., 2022a ) employs supplementary distilled data on both server and client ends, leveraging knowledge distillation to mitigate catastrophic forgetting. Despite effectiveness, most CFL approaches require each client to cache samples locally, a strategy referred to as data rehearsal. Maintaining the local information on the client side is controversial. On the one hand, clients are often limited by storage resources (typically edge devices) and are unable to cache sufficient previous samples for replay. On the other hand, privacy concerns are non-negligible in the FL setting. For instance, some companies will collect user data to update models in a short period, but this data may contain timestamps and need to be deleted. Meanwhile, generating synthetic samples for replay using generative models may also pose a risk of privacy leakage. As shown in Table 1 , we provide a detailed analysis of the latest CFL algorithms and the specific weak points involved. Moreover, existing methods often overlook the data heterogeneity in real-world FL scenarios. Simply applying the same techniques across different clients can lead to decreased model performance when dealing with highly heterogeneous distributed data. In this paper, we aim to enhance the regularization technique for continual federated learning by adhering to the following three principles: (1) free rehearsal, (2) low computational cost, and (3) robustness to data heterogeneity. In response to the first two principles mentioned above, we observe that a considerable number of traditional regularization techniques in the centralized environment can offer a possible solution intuitively. To verify it, we deploy these techniques into the CFL scenario and conduct extensive experiments, which proves that some of the traditional regularization techniques can indeed achieve fairish results, where Synaptic Intelligence (SI) ( Zenke et al., 2017 ) is particularly outstanding and greatly outperforms other methods. Despite the advantages of these techniques, we find that directly combining them with CFL fails to maintain stable performance under varying data heterogeneity, with SI experiencing the most significant decline. In principle, this is due to surrogate loss, as the main component of SI, being merely calculated based on the local data distribution, causing the local optimization objective to not align with that of the global data distribution in the case of highly heterogeneous data. To address this issue, we propose the Synergistic Synaptic Intelligence for the continual Federated Learning (FedSSI). Specifically, in FedSSI, each client will calculate the surrogate loss for previous tasks based not only on information from the local dataset but also on its correlation to the global dataset. We introduce a personalized surrogate model (PSM) for each client, which incorporates global knowledge into local caching so that the surrogate loss reflects both local and global understandings of the data. The SI algorithm is then employed to mitigate catastrophic forgetting, with each client training the local model using both the new task loss and the PSM. Through extensive experiments on various datasets and two types of incremental tasks (Class-IL and Domain-IL), we demonstrate that