Decentralized Optimization with Amplified Privacy via Efficient Communication

Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi · 2025

Read Decentralized Optimization with Amplified Privacy via Efficient Communication by Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07102.txt.

Introduction

Wei Huo and co-authors show that efficient communication can itself protect privacy in decentralized learning in a 2025 paper. In decentralized optimization, many agents cooperate without a central server, which raises two worries: heavy communication and leaks of private data. Standard privacy methods add noise, which costs accuracy.

Their algorithm activates agents at random and sparsifies the messages they exchange. Using differential privacy analysis, the authors show these two features reduce the noise needed for the same privacy guarantee. Privacy is amplified while accuracy improves, a rare win-win for multi-agent systems such as sensor networks and federated devices.

Catalog record

CREATORWei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decentralized Optimization with Amplified Privacy via Efficient Communication Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi Abstract Decentralized optimization is crucial for multi-agent systems, with significant concerns about communication efficiency and privacy. This paper explores the role of efficient communication in decentralized stochastic gradient descent algorithms for enhancing privacy preservation. We develop a novel algorithm that incorporates two key features: random agent activation and sparsified communication. Utilizing differential privacy, we demonstrate that these features reduce noise without sacrificing privacy, thereby amplifying the privacy guarantee and improving accuracy. Additionally, we analyze the convergence and the privacy-accuracy-communication trade-off of the proposed algorithm. Finally, we present experimental results to illustrate the effectiveness of our algorithm. I Introduction Decentralized optimization enables multiple agents to collaborate in minimizing a shared objective without a central coordinator. This approach has diverse applications, including large-scale model training in artificial intelligence [ 1 ] and spectrum sensing in cognitive networks [ 2 ] . Various decentralized optimization methods have been developed, such as decentralized stochastic gradient descent (SGD) [ 1 ] , gradient tracking [ 3 ] and alternating direction method of multipliers [ 4 ] . Despite the advancement of decentralized optimization techniques, issues such as communication overload and privacy breaches arise from agents broadcasting messages, especially in large-scale networks with mobile devices. For instance, in GPT-2 [ 5 ] , which comprises approximately 1.5 billion parameters, transmitting these parameter at every iteration would result in prohibitively high communication costs. Additionally, storing data locally on agents does not provide sufficient privacy protection, as demonstrated by recent inference attacks in [ 6 , 7 ] . Therefore, there is a pressing need for integrated algorithms that effectively address communication and privacy challenges. To address the communication bottleneck in decentralized optimization problems, some studies focus on reducing the number of communication rounds. For example, event-triggered methods have been used to balance the trade-off between communication costs and convergence performance [ 8 ] . Other works employ compression techniques, such as quantization methods to decrease transmitted bit count [ 9 ] and sparsification techniques to reduce message size during transmission [ 10 ] . However, these approaches did not explicitly consider privacy preservation. Decentralized optimization with privacy protection increasingly relies on differential privacy (DP) as a leading standard. DP, pioneered by Dwork [ 11 ] , is favored for its privacy assurances and efficiency in data analysis tasks. Huang et al. [ 12 ] introduced a differentially private distributed optimization algorithm by perturbing transmitted data with Laplacian noise. However, this method slowed convergence due to the linearly decaying learning rate. Ding et al. [ 13 ] enhanced the convergence rate to linear based on a gradient tracking scheme, but at the expense of increased communication load due to the necessity for agents to transmit two variables. While stochastic quantization schemes have been employed solely to ensure DP, the DP is established only for each iteration, risking cumulative privacy loss over multiple iterations [ 14 , 15 ] . Liu et al. [ 16 ] developed a class of differentially private distributed dual averaging algorithms and implemented agent subsampling procedures to bolster DP guarantees. For a comprehensive overview, see [ 17 ] . Despite these advancements, the applicability of these methodologies is confined to convex optimizations, whereas many practical tasks entail non-convex optimization problems, such as neural network configurations in deep learning. Previous research has predominantly focused on either saving communication costs or preserving privacy in decentralized convex optimization [ 18 , 8 , 9 , 10 , 12 , 13 , 16 ] , we delve into decentralized non-convex optimization problems under joint communication and DP constraints. Recent works by Xie et al. [ 19 , 20 ] integrated compression and DP for decentralized optimization. However, their approaches treating communication reduction and perturbation as independent operations led to cumulative errors, degrading optimization accuracy over finite iterations. For a centralized and convex setup, Farokhi [ 21 ] investigated the impact of gradient sparsification on DP, and developed differentially private SGD with sparsification that achieves a better privacy-accuracy tradeoff, provided that the privacy budget is tight enough. In the federated setting, some studies used compression during uplink transmission to enhance privacy [ 22 , 23 ] . However, Chen at al. [ 23 ] only considered central DP, which is weaker than agent-level privacy. Additionally, communication in decentralized networks is much more complicated than in master-slave settings. To address these challenges, we aim to investigate the interplay between communication efficiency and DP. Integrating communication reduction with noise perturbation to enhance privacy and quantifying this enhancement present ongoing challenges. To overcome these obstacles, we propose a decentralized optimization algorithm based on the framework shown in Fig. 1 . Fig. 1 : DP amplified by random activation and sparsification. In our approach, agents are randomly activated during each iteration, with only active agents adding noise to their update, followed by message sparsification. This framework improves communication efficiency by reducing communication rounds and transmitted data size. Random agent activation ensures that only a randomly selected subset of data samples contributes to gradient calculations, thus improving privacy protection. Sparsification also reduces gradient sensitivity, leading to decreased privacy loss in each communication round. Our method allows for reducing noise intensity without compromising DP, achieving a better balance between privacy, accuracy, and communication efficiency. Theoretical analysis is provided to demonstrate the convergence of our approach and its privacy guarantee. The main contributions of our work are summarized as follows: 1) We introduce a novel differentially private decentralized non-convex optimization algorithm that incorporates random agent activation and message sparsification ( Algorithm 1 ). Unlike previous studies that focused solely on communication reduction [ 18 , 8 , 9 , 10 ] or privacy preservation [ 12 , 13 , 16 ] , our algorithm simultaneously enhances both communication efficiency and privacy preservation. Additionally, in contrast to [ 22 , 23 ] , our approach does not require a central node for global information aggregation. 2) We present a rigorous DP analysis of the designed algorithm ( Theorem 1 ), quantifying the amplified privacy level attributed to random activation and sparsification. Our findings indicate that reducing the agent activation probability and the number of coordinates during sparsification can lower the required noise for achieving (\varepsilon,\delta) -DP, unlike the works of Xie et al. [ 19 , 20 ] which overlook the interplay between compression and privacy preservation. To our knowledge, this is the first study to analyze how efficient communication impacts privacy amplification in decentralized non-convex optimization. 3) We establish the non-asymptotic convergence of our algorithm in finding first-order stationary points and demonstrate a better trade-off between privacy, accuracy, and communication efficiency ( Theorem 2 , Corollary 1 ). The paper is organized as follows. Section II lays out the preliminaries and problem formulations. In Section III , we propose a decentralized optimization algorithm with amplified DP. Section IV discusses the DP analysis and Section V delves into the convergence analysis. Section VI presents numerical simulations to illustrate our findings. Finally, Section VII covers conclusions and future research. Notation: Let \mathbb{R}^{p} and \mathbb{R}^{p\times q} denote p -dimensional vectors and p\times q -dimensional matrices, respectively. I_{p}\in\mathbb{R}^{p\times p} represents a

Read the complete text