A New Federated Learning Framework Against Gradient Inversion Attacks
Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, Liangqiong Qu · 2024
Read A New Federated Learning Framework Against Gradient Inversion Attacks by Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, Liangqiong Qu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07187.txt.
Introduction
Shared gradients can leak private data, and a 2024 paper by Pengxin Guo and co-authors redesigns federated learning to cut that leak. In federated learning, phones and hospitals train a shared model without handing over raw data, yet attackers can rebuild inputs from the gradients that do get shared.
Their framework, HyperFL, inserts a hypernetwork that generates each local model's settings. Only the generator's own parameters travel to the server for merging, breaking the direct line between shared numbers and private data. Older defenses, from secret sharing to added noise, all traded accuracy for safety. The authors back the design with theory and pitch it as privacy without the usual heavy cost.
Catalog record
| CREATOR | Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, Liangqiong Qu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07187
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A New Federated Learning Framework Against Gradient Inversion Attacks Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, Liangqiong Qu Abstract Federated Learning (FL) aims to protect data privacy by enabling clients to collectively train machine learning models without sharing their raw data. However, recent studies demonstrate that information exchanged during FL is subject to Gradient Inversion Attacks (GIA) and, consequently, a variety of privacy-preserving methods have been integrated into FL to thwart such attacks, such as Secure Multi-party Computing (SMC), Homomorphic Encryption (HE), and Differential Privacy (DP). Despite their ability to protect data privacy, these approaches inherently involve substantial privacy-utility trade-offs. By revisiting the key to privacy exposure in FL under GIA, which lies in the frequent sharing of model gradients that contain private data, we take a new perspective by designing a novel privacy preserve FL framework that effectively “breaks the direct connection” between the shared parameters and the local private data to defend against GIA. Specifically, we propose a Hypernetwork Federated Learning (HyperFL) framework that utilizes hypernetworks to generate the parameters of the local model and only the hypernetwork parameters are uploaded to the server for aggregation. Theoretical analyses demonstrate the convergence rate of the proposed HyperFL, while extensive experimental results show the privacy-preserving capability and comparable performance of HyperFL. Code is available at https://github.com/Pengxin-Guo/HyperFL. 1 Introduction Deep Neural Networks (DNN) have achieved remarkable success on a variety of computer vision tasks, relying on the availability of a large amount of training data ( Krizhevsky, Sutskever, and Hinton 2012 ; He et al. 2016 ; Dosovitskiy et al. 2021 ; Liu et al. 2021 ; Wang et al. 2023 ) . However, in many real-world applications, training data is distributed across different institutions, and data sharing between these entities is often restricted due to privacy and regulatory concerns. To alleviate these concerns, Federated Learning (FL) ( McMahan et al. 2017 ; Li et al. 2020a ; Qu et al. 2022 ; Zeng et al. 2024 ; Guo et al. 2024 ; Zhang et al. 2024 ) has emerged as a promising approach that enables collaborative and decentralized training of AI models across multiple institutions without sharing of personal data externally. Figure 1: Left . Existing methods mainly explore defenses mechanisms on the shared gradients. Such mechanisms, including SMC, HE, and DP, inherently involve substantial privacy-utility trade-offs. Right . A novel FL framework that “breaks the direct connection” between the shared parameters and the local private data is proposed to achieve a favorable privacy-utility trade-off. Despite the privacy-preserving capability introduced by FL, recent works ( Geiping et al. 2020 ; Huang et al. 2021 ; Hatamizadeh et al. 2023 ) have revealed that FL models are vulnerable to Gradient Inversion Attacks (GIA) ( Fredrikson, Jha, and Ristenpart 2015 ; Zhu, Liu, and Han 2019 ) . GIA can reconstruct clients’ private data from the shared gradients, undermining FL’s privacy guarantees. To remedy this issue, various defense mechanisms have been integrated into FL, including Secure Multi-party Computing (SMC) ( Yao 1982 ; Bonawitz et al. 2017 ; Mugunthan et al. 2019 ; Mou et al. 2021 ; Xu et al. 2022 ) , Homomorphic Encryption (HE) ( Gentry 2009 ; Zhang et al. 2020a ; Zhang et al. 2020b ; Ma et al. 2022 ; Park and Lim 2022 ) and Differential Privacy (DP) ( Dwork 2006 ; Geyer, Klein, and Nabi 2017 ; McMahan et al. 2018 ; Yu, Bagdasaryan, and Shmatikov 2020 ; Bietti et al. 2022 ; Shen et al. 2023 ) . These approaches primarily rely on existing defense mechanisms to enhance privacy protection against GIA without altering the FL framework, as illustrated in the left part of Figure 1 . However, such mechanisms, including SMC, HE, and DP, inherently involve substantial privacy-utility trade-offs. For example, SMC and HE, while providing strong security guarantees through encrypted information exchange, entail high computation and communication costs, making them unsuitable for DNN models with numerous parameters ( Bonawitz et al. 2017 ; Zhang et al. 2020b ) . Although DP approaches are easier to implement, they often fall short in providing sufficient model protection while preserving accuracy ( Geyer, Klein, and Nabi 2017 ; McMahan et al. 2018 ) . These trade-offs within current defense algorithms have inspired us to explore alternative privacy-preserving methods that strike a better balance between privacy and utility, leading to the central question of this paper: Can we design a novel FL framework that offers a favorable privacy-utility trade-off against GIA without relying on existing defense mechanisms? To this end, we revisit the key to privacy exposure in FL under GIA, which lies in the frequent sharing of model gradients that contain private data. Most efforts aim at exploring various advanced defenses mechanisms on the shared gradients to enhance privacy preservation in FL, as shown in the left part of Figure 1 . In contrast, we take a new perspective striving for designing a novel privacy preserve FL framework that “breaks the direct connection” between the shared parameters and the local private data to defend against GIA. In order to achieve this, we explore the potential of hypernetworks, a class of deep neural networks that generate the weights for another network ( Ha, Dai, and Le 2017 ) , as a promising solution, as illustrated in the right part of Figure 1 and Figure 2 . Specifically, we introduce a novel Hypernetwork Federated Learning (HyperFL) framework that adopts a dual-pronged approach—network decomposition and hypernetwork sharing— to “break the direct connection” between the shared parameters and the local private data, as shown in Figure 2 . In light of recent findings regarding minimal discrepancies in feature representations and considerable diversity in classifier heads among FL clients ( Collins et al. 2021 ; Xu, Tong, and Huang 2023 ; Shen et al. 2023 ) , we decompose each local model into a shared feature extractor and a private classifier, enhancing performance in heterogeneous settings and mitigating privacy leakage risks. To further strengthen privacy preservation, we employ an auxiliary hypernetwork that generates feature extractor parameters based on private client embeddings. Instead of directly sharing the feature extractor, only the hypernetwork parameters are uploaded to the server for aggregation, while classifiers and embeddings are trained locally. This auxiliary hypernetwork sharing strategy “breaks the direct connection” between shared parameters and local private data, maintaining privacy while enabling inter-client interaction and information exchange. Remarkably, the design of HyperFL is flexible and scalable, catering to a diverse range of FL demands through its various configurations. We present two major configurations of HyperFL: (1) Main Configuration HyperFL, suitable for simple tasks with small networks, learns the entire feature extractor parameters directly (see Figure 2 ); (2) HyperFL for Large Pre-trained Models (denoted as HyperFL-LPM) targets for complex tasks by using pre-trained models as fixed feature extractors and generating trainable adapter parameters via a hypernetwork ( Houlsby et al. 2019 ) (see Figure 3 ). Both theoretical analysis and extensive experimental results demonstrate that HyperFL effectively preserves privacy under GIA while achieving comparable results and maintaining a similar convergence rate to FedAvg ( McMahan et al. 2017 ) . We hope that the proposed HyperFL framework can encourage the