DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning
Jay Chooi, Kevin Cong, Russell Li, Lillian Sun · 2025
Read DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning by Jay Chooi, Kevin Cong, Russell Li, Lillian Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07843.txt.
Introduction
Jay Chooi, Kevin Cong, and Russell Li bring a favorite optimizer into private training. Differential privacy protects people inside datasets, but private optimizers often lag in accuracy. AdamW trains modern models well in public settings, so a private version is worth building carefully.
DP-AdamW adds privacy noise to decoupled weight decay training. A bias-corrected variant, DP-AdamW-BC, repairs the second-moment estimate that noise distorts. Theory and experiments check whether performance survives the privacy budget. Hospitals, banks, and platforms all need models trained on sensitive data; optimizers that keep both accuracy and formal privacy widen what can be learned safely.
Catalog record
| CREATOR | Jay Chooi, Kevin Cong, Russell Li, Lillian Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning Jay Chooi, Kevin Cong, Russell Li, Lillian Sun Abstract As deep learning methods increasingly utilize sensitive data on a widespread scale, differential privacy (DP) offers formal guarantees to protect against information leakage during model training. A significant challenge remains in implementing DP optimizers that retain strong performance while preserving privacy. Recent advances introduced ever more efficient optimizers, with AdamW being a popular choice for training deep learning models because of strong empirical performance. We study DP-AdamW and introduce DP-AdamW-BC , a differentially private variant of the AdamW optimizer with DP bias correction for the second moment estimator. We start by showing theoretical results for privacy and convergence guarantees of DP-AdamW and DP-AdamW-BC. Then, we empirically analyze the behavior of both optimizers across multiple privacy budgets ( ϵ = 1 , 3 , 7 \epsilon=1,3,7 ). We find that DP-AdamW outperforms existing state-of-the-art differentially private optimizers like DP-SGD, DP-Adam, and DP-AdamBC, scoring over 15% higher on text classification, up to 5% higher on image classification, and consistently 1% higher on graph node classification. Moreover, we empirically show that incorporating bias correction in DP-AdamW (DP-AdamW-BC) consistently decreases accuracy, in contrast to the improvement of DP-AdamBC improvement over DP-Adam. † † affiliationnotice: \icmlEqualContributionAlphabetical 1 Introduction In recent years, deep learning has achieved widespread adoption, with applications ranging from natural language processing to image generation ( 4 ; 10 ) . However, models trained on large and sensitive datasets have been shown to be vulnerable to privacy attacks on their training data, raising significant privacy conerns ( 6 ; 5 ; 2 ) . Differential Privacy (DP) provides a rigorous mathematical framework to address these concerns, enabling the development of algorithms with provable guarantees against leaking individual-specific information ( 9 ) . In stochastic optimization, this has been most commonly realized by the DP-SGD algorithm introduced by 1 . The first application of DP in the deep learning model training pipeline, this approach adds Gaussian noise of variance \sigma^{2}C^{2} after clipping per-sample gradients to a radius C . While DP-SGD offers strong theoretical guarantees, its empirical performance, as with vanilla SGD, often lags behind the adaptive Adam family of optimizers that dominate non-private deep learning. Subsequent work therefore lifted Adam into the private setting; specific differentially private optimizers that were developed to potentially improve the privacy-utility tradeoff include DP-GD, DP-RMSprop, and DP-Adam ( 18 ; 13 ) . Motivated by this line of work, we study differentially private variants of the AdamW optimizer which has been empirically shown to achieve improved generalization performance over Adam, particularly enabling competitive performance with SGD on image classification tasks ( 15 ) . To provide a differentially private implementation of AdamW, we introduce the DP-AdamW algorithm by decoupling the weight decay from the gradient update in the DP-Adam algorithm ( 16 ) . Additionally, we introduce DP-AdamW-BC , a variant of DP-AdamW that corrects for the DP bias in the second moment estimator, which 16 characterized and corrected for DP-Adam, resulting in DP-AdamBC. We provide theoretical results on the privacy and convergence guarantees of the new optimizers and in particular show that DP-AdamW maintains similar privacy bounds and convergence guarantees to DP-AdamBC. We then compare the performance of our DP-AdamW and DP-AdamW-BC to previous optimizers, empirically finding that they consistently improve on the privacy-accuracy tradeoff compared to their DP-Adam counterparts. In contrast to the findings of 16 , we show that incorporating bias correction in DP-AdamW-BC consistently decreases accuracy across diverse tasks. We demonstrate that DP-AdamW outperforms DP-SGD even in image classification, for which Adam is known to not generalize as well ( 15 ) . The remainder of our paper proceeds as follows. In Section 3 , we present the DP-AdamW and DP-AdamW-BC algorithms and their theoretical guarantees. In Section 4 , we detail our experiments and present the results of using the DP-AdamW and DP-AdamW-BC optimizers in comparison to DP-Adam and DP-AdamW. In Section 5 , we discuss these results, and in Sections F and 6 , we conclude and offer further avenues of work. 2 Motivation and Related Work Differentially private optimization was first studied by ( 1 ) , who introduced Differentially Private Stochastic Gradient Descent (DP-SGD). DP-SGD achieved provable privacy guarantees via adding noise to the gradients during gradient descent. However, the additional noise led to significantly worse performance in comparison to non-private models. This led to the work of ( 14 ) , who showed that finetuning pretrained models under DP-Adam achieved strong performance on par with non-private models. Moreover, they demonstrated a scaling law of DP-models by parameters and empirically invalidated the hypothesis that DP-ML suffers from dimension-dependent performance degradation. This set the stage for practical applications of DP on deep learning. More recently, ( 16 ) proposed an improvement to DP-Adam, namely DP-AdamBC, adding a correction term in DP-Adam to remove the bias in the second moment estimate arising from the added gaussian noise. DP-AdamBC achieved an improvement over DP-Adam across several different target privacy budgets on image, text and graph node classification tasks. Motivated by these developments, our present work aims to improve the performance of DP-Adam and DP-AdamBC via adding weight decay, thereby yielding DP-AdamW and DP-AdamW-BC. We finally evaluate the performance of these optimizers using the same suite of tasks in their paper. 3 DP-AdamW: Algorithm and Theoretical Guarantees In this section, we introduce the DP-AdamW algorithm and its variant, DP-AdamW-BC, and present theoretical results on their privacy and convergence guarantees 3.1 DP-AdamW: The Algorithm We now introduce the DP-AdamW algorithm. The main point is to add Gaussian noise to the gradients in the AdamW algorithm. Given a batch of gradients g_{i} , we compute the average gradient \overline{g} . Define the noised gradient \tilde{g}=\frac{1}{B}\sum_{i}\frac{g_{i}}{\max(1,\frac{\|g_{i}\|_{2}}{C})}+\frac{1}{B}\mathcal{N}(0,\sigma^{2}C^{2}\mathbf{I}). We then use the noised gradient \tilde{g} as the input gradient into the AdamW algorithm of ( 15 ) . 1 1 1 Alternatively, one can view this as adding decoupled weight decay into the DP-Adam algorithm of ( 16 ) The resulting DP-AdamW algorithm is given in Algorithm 1 . Algorithm 1 DP-AdamW 1: total steps T , learning-rate schedule \{\eta_{t}\}_{t=1}^{T} , clip norm C , weight-decay \lambda , hyper-parameters \alpha,\,\beta_{1},\,\beta_{2} , noise multiplier \sigma , initial parameters \theta_{0} , numerical stability constant \epsilon_{0} 2: m_{0}\leftarrow 0 , v_{0}\leftarrow 0 3: for t\leftarrow 1 to T do 4: g_{t}\leftarrow\nabla f\!\bigl(\theta_{t-1}\bigr) 5: \displaystyle\tilde{g}_{t}\leftarrow\frac{1}{B}\sum_{i}\frac{g_{i}}{\max\left(1,\frac{\|g_{i}\|_{2}}{C}\right)}\;+\;\frac{1}{B}\,\mathcal{N}\!\bigl(0,\sigma^{2}C^{2}I\bigr) 6: m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})\,\tilde{g}_{t} 7: v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})\,\tilde{g}_{t}^{\,2} 8: \hat{m}_{t}\leftarrow\frac{m_{t}}{1-\beta_{1}^{\,t}} 9: \hat{v}_{t}\leftarrow\frac{v_{t}}{1-\beta_{2}^{\,t}} 10: \displaystyle\theta_{t}\leftarrow\theta_{t-1}-\eta_{t}\left(\frac{\hat{m}_{t}}{\sqrt{\hat{v}_{t}+\epsilon_{0}}}+\lambda\,\theta_{t-1}\right) 11: end for Moreover, ( 16 ) observe that within the Adam update, the estimate of the second moment is biased due to the use of a noised gradient. They offer a method of bias-correction, in particular by replacing the update term \frac{\hat{m}_{t}}{\sqrt{\hat{v}_{t}}} with \frac{\hat{m}_{t}}{\sqrt{\hat{v}_{t}-\left(\frac{\sigma C}{B}\right)^{2}}}=\frac{\hat{m}_{t}}{\sqrt{\hat{v}_{t}-\Phi}}. We also consider this modification, which is given in Algorithm 2 . Algorithm 2 DP-AdamW-BC 1: total steps T , learning-rate schedule \{\eta_{t}\}_{t=1}^{T} , clip norm C , weight-decay \lambda , hyper-parameters \alpha,\,\beta_{1},\,\beta_{2} , noise multiplier \sigma , initial parameters \theta_{0} , numerical stability constant \gamma 2: m_{0}\leftarrow 0 , v_{0}\leftarrow 0 3: for t\leftarrow 1 to T do 4: g_{t}\leftarrow\nabla f\!\bigl(\theta_{t-1}\bigr) 5: \displaystyle\tilde{g}_{t}\leftarrow\frac{1}{B}\sum_{i}\frac{g_{i}}{\max\left(1,\frac{\|g_{i}\|_{2}}{C}\right)}\;+\;\frac{1}{B}\,\mathcal{N}\!\bigl(0,\sigma^{2}C^{2}I\bigr) 6: m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})\,\tilde{g}_{t} 7: v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})\,\tilde{g}_{t}^{\,2} 8: \hat{m}_{t}\leftarrow\frac{m_{t}}{1-\beta_{1}^{\,t}} 9: \hat{v}_{t}\leftarrow\frac{v_{t}}{1-\beta_{2}^{\,t}} 10: \displaystyle\theta_{t}\leftarrow\theta_{t-1}-\eta_{t}(\frac{\hat{m}_{t}}{\sqrt{\max(\hat{v}_{t}-(\frac{\sigma C}{B})^{2},\gamma)}}+\lambda\,\theta_{t-1}) 11: end for 3.2 Privacy Guarantees