Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data

Abhijit Chunduru, Majid Morafah, Mahdi Morafah, Vishnu Pandi Chellapandi, Ang Li · 2025

Read Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data by Abhijit Chunduru, Majid Morafah, Mahdi Morafah, Vishnu Pandi Chellapandi, Ang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20485.txt.

Introduction

Chunduru and four co-authors addressed a flaw in federated learning in this paper from twenty twenty-five. In federated learning, many devices train one shared model without pooling their data. When each device's data look different, local training pulls the model toward local habits. The team probed how this warps the shared decision boundary.

A toy example showed clients forget the global boundary even when they start from the best weights. Their fix, FedProj, anchors local updates with stored average outputs from an ensemble. A server-side loss then re-tunes the fused model. Public unlabeled data supply the shared reference points. In tests the method beat leading rivals by wide margins. The authors say preserving global knowledge is the key step.

Catalog record

CREATORAbhijit Chunduru, Majid Morafah, Mahdi Morafah, Vishnu Pandi Chellapandi, Ang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data Abhijit Chunduru, Majid Morafah, Mahdi Morafah, Vishnu Pandi Chellapandi, Ang Li Abstract The inevitable presence of data heterogeneity has made federated learning very challenging. There are numerous methods to deal with this issue, such as local regularization, better model fusion techniques, and data sharing. Though effective, they lack a deep understanding of how data heterogeneity can affect the global decision boundary. In this paper, we bridge this gap by performing an experimental analysis of the learned decision boundary using a toy example. Our observations are surprising: (1) we find that the existing methods suffer from forgetting and clients forget the global decision boundary and only learn the perfect local one, and (2) this happens regardless of the initial weights, and clients forget the global decision boundary even starting from pre-trained optimal weights. In this paper, we present FedProj, a federated learning framework that robustly learns the global decision boundary and avoids its forgetting during local training. To achieve better ensemble knowledge fusion, we design a novel server-side ensemble knowledge transfer loss to further calibrate the learned global decision boundary. To alleviate the issue of learned global decision boundary forgetting, we further propose leveraging an episodic memory of average ensemble logits on a public unlabeled dataset to regulate the gradient updates at each step of local training. Experimental results demonstrate that FedProj outperforms state-of-the-art methods by a large margin. 1 Introduction Federated Learning (FL) [ 20 ] has emerged as a privacy-preserving framework that trains a shared global model across multiple clients without exchanging raw data, under the coordination of a central server. The most widely adopted FL method, FedAvg [ 32 ] , aggregates local weight or gradient updates to learn the global model. Although FedAvg has demonstrated promise across various applications, its performance is significantly hindered by data heterogeneity [ 13 , 45 , 28 , 4 , 3 , 5 , 6 , 35 , 36 , 34 ] among clients—commonly referred to as Non-IID data—which can lead to poor convergence and degraded performance. Numerous studies have sought to address the challenges posed by Non-IID data in FL by mitigating client drift [ 26 , 21 ] , enhancing server-side model fusion and distillation [ 7 , 51 , 24 ] , and refining local training protocols [ 9 ] . Despite these efforts, a critical issue remains underexplored: during local training, clients tend to overfit to their individual objectives, thereby catastrophically forgetting the global knowledge and decision boundaries learned by the shared aggregated model . Although prior works have acknowledged this phenomenon [ 47 ] , a comprehensive experimental investigation quantifying global knowledge forgetting is lacking. In our work, we first present an empirical pilot study of standard federated learning, i.e. FedAvg, under Non-IID settings, demonstrating that local clients often completely forget global knowledge and converge to models that are fine-tuned solely to their local objectives. Consequently, averaging such divergent models merely restores the original performance rather than enhancing it , and the server-side fusion is further compromised as client models have become increasingly dissimilar. Notably, this catastrophic forgetting occurs even when clients begin training from a pre-trained global model. Motivated by these insights we propose a novel federated learning method called FedProj . Our approach mitigates global knowledge forgetting by imposing explicit constraints on local gradient updates where the actual forgetting happens, thereby preserving the learned global knowledge. Specifically, we maintain a small episodic memory of global knowledge using a public dataset and formulate an optimization problem that constrains new gradient directions to prevent an increase in the losses associated with the global knowledge. At the server side, we further enhance model fusion through ensemble knowledge distillation using both logits and feature representations, and we incorporate a novel weight divergence regularization term to alleviate the adverse effects of noisy distillation. Extensive experiments on both computer vision and natural language processing tasks demonstrate that FedProj significantly outperforms state-of-the-art methods across multiple datasets, achieving superior performance in Non-IID federated learning scenarios. Contributions. Our work makes the following contributions: • We present the first empirical pilot study that provides new insights into the phenomenon of catastrophic forgetting in standard federated learning under data heterogeneity. • We introduce a novel federated learning method, FedProj , which mitigates global knowledge forgetting by imposing explicit constraints on local gradient updates and preserving global knowledge. • We conduct extensive experiments on both CV and NLP tasks and demonstrate that our proposed method outperforms state-of-the-art approaches on various benchmarks and datasets under Non-IID settings. Organization. The remainder of this paper is organized as follows. In Section 2, we review the related work. Section 3 presents our empirical pilot study. In Section 4, we describe the proposed FedProj methodology. Section 5 details our experimental results, and Section 6 concludes the paper. 2 Related Works Continual Learning. In continual learning, gradient projection methods mitigate catastrophic forgetting by constraining parameter updates to preserve past knowledge. These approaches identify critical directions in weight space and restrict updates to be orthogonal or complementary to them [ 48 , 12 ] . In particular, Gradient Projection Memory (GPM) [ 38 ] focuses on key gradient subspaces and explicitly preserves the past gradient subspaces. More recent methods relax strict orthogonality to balance stability and plasticity [ 30 , 37 ] . These methods frame continual learning as a stability-plasticity tradeoff in linear algebra, ensuring knowledge retention while allowing adaptability. Our work draws intuition from these body of works in continual learning to allieviate the catastrophic forgetting in FL. FL with Non-IID Data. Federated learning with non-IID data presents significant challenges, prompting diverse strategies to improve convergence and accuracy. FedAvg [ 32 ] established the foundation for FL by averaging local model updates, but it struggles with statistical heterogeneity. To address this, FedProx [ 26 ] introduces a proximal term to stabilize local updates, while FedNova [ 42 ] normalizes local contributions, mitigating objective inconsistency. SCAFFOLD [ 21 ] introduces control variates to correct local updates, effectively reducing drift by anchoring clients to a global direction. FedGen [ 50 ] tackles non-IID challenges by generating synthetic data at the server using generative models, improving generalization across clients. For personalization, Ditto [ 27 ] maintains dual local-global models, improving client-specific performance, while pFedHN [ 40 ] uses hypernetworks to generate personalized models, enhancing adaptability. FedBN [ 29 ] addresses feature shift by keeping batch normalization layers local, while sharing other parameters, boosting robustness in non-IID settings. MOON [ 25 ] employs contrastive learning to align local and global models, reducing divergence. Comparison with Close Works. FedDF [ 31 ] enhances server-side performance through knowledge distillation, refining the global model by distilling knowledge from local models. FedET [ 8 ] employs ensemble distillation, combining local knowledge into a generalized global model, improving accuracy on heterogeneous data. FedGKT [ 17 ] introduces group knowledge transfer, where lightweight client models distill knowledge into a larger server model, boosting efficiency and handling model heterogeneity. Researchers in [ 15 ] propose a privacy-preserving FL framework with one-shot offline knowledge distillation using unlabeled public data, reducing communication overhead while enhancing privacy guarantees. In contrast, FedProj directly

Read the complete text