DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm

Junkang Liu, Yuxuan Tian, Fanhua Shang, Senior Member, IEEE, Hongying Liu, Senior Member, IEEE · 2025

Read DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm by Junkang Liu, Yuxuan Tian, Fanhua Shang, Senior Member, IEEE, Hongying Liu, Senior Member, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27504.txt.

Introduction

Junkang Liu leads six co-authors in DP-FedPGN, a differentially private federated learning algorithm that seeks globally flat minima. Client-level differential privacy prevents inference attacks and leakage, but private training usually sharpens the loss landscape and hurts generalization. Sharpness-aware methods find locally flat spots, which may not reflect global flatness under privacy noise.

Their algorithm adds a global gradient norm penalty to the local loss, steering toward a globally flat minimum while also shrinking local update norms and thus clipping error. Theory analyzes how the penalty mitigates privacy-driven degradation, removes heterogeneity impact, and keeps fast convergence, with strict Renyi differential privacy guarantees. Tests on image and language tasks with ResNet and transformer models improve significantly over state-of-the-art private methods.

Catalog record

CREATORJunkang Liu, Yuxuan Tian, Fanhua Shang, Senior Member, IEEE, Hongying Liu, Senior Member, IEEE
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm Thanks: Fanhua Shang, Junkang Liu, and Daorui Ding, Junchao Zhou are with the School of Computer Science and Technology, College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China. E-mail: [email protected] Liu is with the Medical College, Tianjin University, Tianjin, 300072, China. E-mail: [email protected] Liu are with the School of Artificial Intelligence, Xidian University, China. E-mail: [email protected]. Junkang Liu, Yuxuan Tian, Fanhua Shang, Senior Member, IEEE, Hongying Liu, Senior Member, IEEE Abstract To prevent inference attacks in Federated Learning (FL) and reduce the leakage of sensitive information, Client-level Differentially Private Federated Learning (CL-DPFL) is widely used. However, current CL-DPFL methods usually result in sharper loss landscapes, which leads to a decrease in model generalization after differential privacy protection. By using Sharpness Aware Minimization (SAM), the current popular federated learning methods are to find a local flat minimum value to alleviate this problem. However, the local flatness may not reflect the global flatness in CL-DPFL. Therefore, to address this issue and seek global flat minima of models, we propose a new CL-DPFL algorithm, DP-FedPGN, in which we introduce a global gradient norm penalty to the local loss to find the global flat minimum. Moreover, by using our global gradient norm penalty, we not only find a flatter global minimum but also reduce the locally updated norm, which means that we further reduce the error of gradient clipping. From a theoretical perspective, we analyze how DP-FedPGN mitigates the performance degradation caused by DP. Meanwhile, the proposed DP-FedPGN algorithm eliminates the impact of data heterogeneity and achieves fast convergence. We also use Rényi DP to provide strict privacy guarantees and provide sensitivity analysis for local updates. Finally, we conduct effectiveness tests on both ResNet and Transformer models, and achieve significant improvements in six visual and natural language processing tasks compared to existing state-of-the-art algorithms. The code is available at https://github.com/junkangLiu0/DP-FedPGN I Introduction (a) DP-FedAvg (b) DP-FedSAM (c) DP-FedPGN (d) Low heterogeneity (e) high heterogeneity (f) DP-FedPGN Fig. 1 : The global loss surface for DP-FedAvg [ 1 ] , DP-FedSAM [ 2 ] , and the proposed DP-FedPGN method with ResNet-18 on CIFAR100 in the case of Dirichlet-0.1. Federated learning (FL) [ 3 ] enables the training of collaborative models among distributed clients without sharing raw data. Although FL algorithms aim to protect privacy, privacy leakage remains a critical concern [ 4 , 5 , 6 , 7 , 8 ] , as adversaries can infer sensitive client information through model updates. Client-level Differentially Private Federated learning (CL-DPFL) [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] , has emerged as a practical standard to safeguard the participation of individual clients, which ensures that adversaries cannot discern whether a client user contributed to training by observing the aggregated model updates [ 18 , 19 , 20 ] . Although differential privacy at the client level provides strong privacy guarantees, it can lead to significant performance degradation of existing FL methods. This phenomenon stems from two critical mechanisms: (1) Gradient clipping bounds the norm of local updates to control sensitivity. This operation inevitably discards valuable information beyond the threshold, particularly harming models trained on non-IID data, where updates exhibit high variability [ 9 ] . (2) The Gaussian noise scaled by the clipping norm introduces variance across clients, which exacerbates model inconsistency in heterogeneous data settings and damages the accuracy of the global model [ 21 ] . (a) DP-FedSAM-Global (b) DP-FedSAM-Local (c) DP-FedPGN-Global (d) DP-FedPGN-Local Fig. 2 : Loss landscapes of global and local models trained by DP-FedSAM and DP-FedPGN with ResNet-18 on CIFAR100 in the case of Dirichlet-0.1. DP-FedPGN global model is flatter than DP-FedSAM global model. A lot of work has been done to address these two issues, and Fed-SMP [ 22 ] mitigates the impact of noise on model updates through sparse updates. DP-FedAvg-BLUR [ 9 ] reduces the threshold for local model variation by adding bounded local update regularization to the loss function. However, these methods have not fundamentally solved the problem of severe performance degradation in CL-DPFL models. Our extensive experiments demonstrate that the composite effect of gradient clipping and Gaussian noise disrupts the geometry of the loss landscape, trapping the model in sharp minima in Figure 1 , which is the fundamental reason for the severe performance degradation of CL-DPFL models. Numerous studies [ 23 , 24 ] have shown that sharp minima can reduce the model’s generalization ability and robustness. The latest method, DP-FedSAM [ 2 ] , attempts to flatten local loss landscapes by locally using Sharpness-Aware Minimization (SAM) [ 25 ] , thereby mitigating the model’s reduced generalization caused by DP noise. However, DP-FedSAM mainly focuses on reducing local sharpness without aligning local and global flatness objectives. We find through extensive experiments that in DPFL, the loss landscape flatness of local models may not reflect the loss landscape flatness of the global model after server aggregation in DP-FedSAM, resulting in the aggregated model being located in sharp areas (see Figure 1 (b) and 2 (a, b)). To address this issue, we impose an additional penalty on the gradient norm of the loss function of the global model. The motivation for penalizing the gradient norm of the global loss function is to encourage the optimizer to find the minima located in a relatively flat neighborhood region, as such flat minima have been shown to achieve better model generalization than sharp minima [ 23 , 24 ] . Figure 4 provides a simple example that intuitively illustrates the relationship between gradient norm and minimum flatness. Previous literature [ 26 ] has shown that the model is flatter at the minima where the gradient norm is smaller. However, in federated learning, accessing the gradient of the global model is impractical, so we use the global variation of the model to estimate the gradient of the global model. We refer to this algorithm as differentially private federated learning via global penalizing gradient norm (DP-FedPGN). Through extensive experiments, it has been shown that DP-FedPGN can not only find a flatter global minimum than DP-FedSAM and DP-FedAvg in Figure 1 (b,c), 2 , and 3 , but also lower the threshold for local model variation in Figure 6 . In addition, Laplacian smoothing (LS) has been widely proven to reduce the noise of gradients, such as in stochastic gradient descent (SGD) [ 27 ] . Laplacian smoothing can also reduce the Gaussian noise introduced by DP [ 28 , 29 ] . Moreover, our experiment reveals that Laplacian smoothing can also make the model’s minima flatter in Figure 8 . Therefore, we add Laplacian smoothing to DP-FedPGN and proposed the DP-FedPGN-LS algorithm. Fig. 3 : Comparison of loss landscapes of DP-FedAvg (up), DP-FedSAM (middle), and DP-FedPGN (down) on CIFAR10 with ResNet-18 in the case of Dirichlet-0.1. I-A Motivation The existing DPFL approaches, including DP-FedAvg [ 1 ] , Fed-SMP [ 22 ] , DP-FedAvg with BLUR and LUS [ 9 ] , and DP-FedSAM [ 2 ] , all suffer from a critical limitation: exacerbated client-side model divergence and convergence to sharp minima in the global

Read the complete text