AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks
Huiqi Zhang [email protected], Fang Xie [email protected] · 2025
Read AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks by Huiqi Zhang [email protected], Fang Xie [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06525.txt.
Introduction
Huiqi Zhang and Fang Xie published a research paper in 2025 on training neural networks with a privacy guarantee while wasting less of the model's skill. Private training clips each update and adds noise, but in big models the noise swamps the signal. Their method first measures which weights matter most, under the same privacy rules.
Training then prunes the least useful weights and clips each remaining one by its own scale. The authors prove the method keeps its privacy promise and still converges. On one digit benchmark it nearly matched a non-private model under a loose budget. On a photo benchmark it beat the non-private baseline outright.
Catalog record
| CREATOR | Huiqi Zhang [email protected], Fang Xie [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06525
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks Huiqi Zhang [email protected], Fang Xie [email protected] Abstract Differential privacy has been proven effective for stochastic gradient descent; however, existing methods often suffer from performance degradation in high-dimensional settings, as the scale of injected noise increases with dimensionality. To tackle this challenge, we propose AdaDPIGU—a new differentially private SGD framework with importance-based gradient updates tailored for deep neural networks. In the pretraining stage, we apply a differentially private Gaussian mechanism to estimate the importance of each parameter while preserving privacy. During the gradient update phase, we prune low-importance coordinates and introduce a coordinate-wise adaptive clipping mechanism, enabling sparse and noise-efficient gradient updates. Theoretically, we prove that AdaDPIGU satisfies ( ε , δ ) (\varepsilon,\delta) -differential privacy and retains convergence guarantees. Extensive experiments on standard benchmarks validate the effectiveness of AdaDPIGU. All results are reported under a fixed retention ratio of 60%. On MNIST, our method achieves a test accuracy of 99.12% under a privacy budget of ϵ = 8 \epsilon=8 , nearly matching the non-private model. Remarkably, on CIFAR-10, it attains 73.21% accuracy at ϵ = 4 \epsilon=4 , outperforming the non-private baseline of 71.12%, demonstrating that adaptive sparsification can enhance both privacy and utility. Huiqi Zhang and Fang Xie † † heading: 00 2025 1- 1/21; Revised 5/22 9/22 21-0000 † † shortheadings: AdaDPIGU: Private SGD with Adaptive Clipping / Zhang and Xie † † firstpage: 1 † † editor: My editor keywords Differential privacy, Deep neural networks, Gradient sparsification, Importance update, Adaptive clipping 1 Introduction Deep learning has achieved remarkable success in tasks such as image recognition Lundervold and Lundervold 2019 ; Zhou et al. 2021 , text analysis Chen et al. 2019 , and recommendation systems Lü et al. 2012 , largely due to centralized training on large-scale datasets. However, this paradigm raises significant privacy concerns, as user data must be transmitted to a central server for model updates. Federated Learning McMahan et al. 2017 addresses this issue by keeping data on local devices and only exchanging model parameters. Nonetheless, studies have shown that model updates alone can leak sensitive information Fredrikson et al. 2015 ; Melis et al. 2019 ; Nasr et al. 2019 ; Phong et al. 2017 ; Song et al. 2017 ; Wang et al. 2019b ; Zhao et al. 2020 , indicating that data localization alone is insufficient for rigorous privacy protection. As illustrated in Figure 1 , deep learning systems are vulnerable to privacy threats in both training and deployment. During training, poisoning attacks can degrade performance and leak data Barreno et al. 2006 , prompting defenses such as data sanitization Rubinstein et al. 2009 and robust learning strategies Biggio et al. 2011 ; Biggio et al. 2009 . During deployment, black-box attacks (e.g., model extraction, membership inference Salem et al. 2018 ; Shokri et al. 2017 ) and white-box attacks (e.g., gradient inversion Zhao et al. 2020 ; Zhu et al. 2019 ) threaten user privacy, with recent works extending such threats to federated settings Geng et al. 2021 ; Li et al. 2021 ; Yin et al. 2021 ; Zhu and Blaschko 2020 . Figure 1 : Deep Learning Pipeline Diagram To address privacy concerns in deep learning, existing approaches mainly follow two directions: cryptographic techniques and random perturbation mechanisms. Cryptographic methods include Homomorphic Encryption (HE) Gentry 2009 ; Gilad-Bachrach et al. 2016 and Secure Multi-Party Computation (MPC) Du and Atallah 2001 ; Hastings et al. 2019 ; Lindell 2005 . While HE allows computation on encrypted data without decryption and MPC enables secure protocols among untrusted parties, both approaches suffer from high computational overhead, limiting their scalability in deep learning. Compared to these methods, Differential Privacy (DP) Dwork 2006 offers formal guarantees by injecting calibrated noise to obscure the influence of any individual data point. With significantly lower computational costs, DP is better suited for large-scale deep learning applications. This thesis focuses on privacy protection—distinct from system security—by preventing the leakage of individual data during training through DP mechanisms. Originally developed for database queries, DP has been successfully adapted to deep learning Shokri and Shmatikov 2015 , with Differentially Private Stochastic Gradient Descent (DPSGD) Abadi et al. 2016 emerging as a leading method. DPSGD applies per-sample gradient clipping and Gaussian noise addition to ensure formal (\varepsilon,\delta) -DP guarantees. However, the injected noise can significantly impair model utility, especially under high-dimensional or long-horizon training regimes. To understand different noise injection strategies, DP mechanisms are typically classified into three categories: (1) output perturbation , which adds noise to the final model Wu et al. 2017 but lacks end-to-end privacy; (2) objective perturbation , suitable for convex problems Chaudhuri et al. 2011 ; Iyengar et al. 2019 ; and (3) gradient perturbation , used in DPSGD, which adds noise during training updates and is the most widely adopted for deep models. Despite its popularity, DPSGD faces challenges. First, it injects uniform noise across all gradient dimensions, which scales with model size and impair convergence Bassily et al. 2014 . Second, the linear accumulation of privacy budget during prolonged training further degrades performance. Additionally, noise is often added to parameters of low importance, thereby weakening the signal for critical gradients Chen et al. 2023 . To investigate the extent of redundancy in gradient dimensions, we performed a statistical analysis on the MNIST dataset. We found that over 99% of gradient values lie within [-0.01,0.01] , indicating strong gradient sparsity. These findings, combined with prior studies on gradient pruning Chen et al. 2020 , suggesting that uniform noise injection is overly redundant. These findings motivate us to design a mechanism that dynamically selects important coordinates and focuses the privacy budget on critical parameters, aiming to enhance both utility and privacy. Figure 2 : Gradient Distribution at Epoch 5 (MNIST) To better appreciate the role of noise injection in DPSGD, we revisit the gradient perturbation pipeline, which consists of the following three steps (see Figure 3(a) ). In the first step (Calculating Gradient), the loss function is differentiated with respect to model parameters during standard SGD to determine update directions. The second step (Clipping Gradient) applies a threshold to rescale gradients that exceed a predefined norm, thereby reducing the impact of individual data samples and mitigating potential privacy leakage. In the final step (Adding Noise), Gaussian noise is added to the clipped gradients, hindering an adversary’s ability to recover the original training data, even if updated gradients are exposed. (a) Standard DPSGD gradient update (b) Importance-based sparse gradient update Figure 3 : Standard DPSGD vs. our AdaDPIGU method for differentially private gradient updates The core contributions of this work are as follows: • We propose AdaDPIGU, a new differentially private optimization framework that combines importance-aware gradient selection with adaptive clipping. Unlike standard DPSGD, which clips and perturbs all gradient dimensions uniformly, AdaDPIGU selectively updates only the most informative coordinates at each iteration. The remaining gradients are zeroed out to avoid injecting noise on redundant directions (see Fig. 3(b) ). This sparsity-aware update mechanism significantly reduces the dimensionality of noise injection and preserves critical optimization signals. • We design an adaptive clipping strategy tailored for sparse gradients. By adjusting the clipping