DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu Abstract Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW’s sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter ( ε , δ ) (\varepsilon,\delta) -DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, ε = 1 \varepsilon=1 ), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available in Appendix. 1 Introduction Figure 1 : Illustration of our DP-FedAdamW. It aggregates the mean of block-wise Bias-Corrected second moment estimates and performs local–global alignment to stabilize DPFL optimization. Figure 2 : An illustration of local update in DP-FedAdamW, which corrects client drift caused through global update guidance. As data volumes surge and privacy concerns grow, centralized learning methods are facing increasing limitations in terms of security and collaboration. Federated Learning (FL) 49 enables decentralized training without sharing raw data, reducing privacy risks 7 . However, FL model updates remain vulnerable to attacks 71 ; 12 ; 72 . Differential Privacy (DP) 11 provides formal protection by adding noise, but this increases gradient variance and hinders convergence, especially under non-IID data 22 . The emergence of large-scale models such as Swin Transformer 47 , BERT 26 , and GPT 2 has become dominant in vision and language tasks. These deep, parameter-heavy models are highly sensitive to optimization settings, which not only affects training stability but also impacts overall performance. Most DPFL algorithms are built upon the SGD optimizer and adopt the DPSGD 1 mechanism for privacy preservation 23 ; 52 ; 73 ; 24 . While effective for small-scale networks, DPSGD becomes inefficient for large Transformer models as the combination of gradient clipping and injected noise severely distorts gradients, amplifying instability and slowing convergence. In the pursuit of more efficient optimizers, adaptive methods like AdamW 48 have become state-of-the-art (SOTA) in deep learning. By decoupling weight decay from adaptive moment estimation, AdamW provides enhanced stability and generalization, making it a compelling choice for various machine learning tasks. These advantages make AdamW a promising candidate for DPFL, where privacy and robustness are crucial. However, our experiments reveal that AdamW fails to deliver its usual benefits in the DPFL setting. In some cases, its performance is comparable to or even worse than SGD. As illustrated in Table 2 , when AdamW is applied locally without any modifications (referred to as DP-LocalAdamW), its performance on the Swin-Tiny model (CIFAR-100, Dirichlet distribution \alpha{=}0.1 ) is worse than DP-FedAvg-LS. These findings suggest that AdamW struggles with the challenges posed by privacy constraints and decentralized FL. Challenges. We identify three main challenges that contribute to the unexpected results. (i) Second-moment estimator variance amplification. Non-IID client data and DP noise jointly inflate the variance of the second-moment estimator, leading to unstable adaptive scaling in AdamW. (ii) Bias in second-moment estimator. Gradient clipping and noise injection introduce a systematic bias in the second-moment estimator of AdamW. (iii) Client crift. Under non-IID data, DP clipping and noise amplify AdamW’s sensitivity to local overfitting, further worsening client drift. All challenges are explored in more detail in Section 4 . These findings raise a key question: How can AdamW maintain effectiveness in the presence of differential privacy and federated heterogeneity? To answer this, we propose DP-FedAdamW, a new differentially private adaptive optimizer that restores the effectiveness of AdamW under non-IID data and privacy noise. DP-FedAdamW (i) aggregates second-moment estimates in a block-wise manner to stabilize variance and improve communication efficiency, (ii) eliminates DP-induced bias through an unbiased second-moment correction, and (iii) enforces local–global update alignment to mitigate client drift, achieving more stable and accurate convergence. Our contributions are summarized as follows: • Empirical importance of AdamW and challenges in DPFL. We empirically demonstrate the effectiveness of AdamW in DPFL. Our analysis reveals three key challenges: high variance in second-moment estimator, bias in second-moment estimator, and high client drift. • DP-FedAdamW: A principled adaptive optimizer for DPFL. We propose DP-FedAdamW, a communication-efficient DPFL algorithm. It integrates global update estimation into local updates to suppress client drift, aggregates block-wise means of second-moment statistics across clients to stabilize variance, and introduces a Bias-Corrected mechanism to eliminate second-moment estimation bias caused by DP. • Theoretical and empirical guarantees. We derive the first convergence guarantee for a DPFL adaptive optimizer that overcomes gradient heterogeneity, achieving a linear speedup rate of \mathcal{O}(\sqrt{(L\Delta\sigma_{l}^{2})/(SKT\epsilon^{2})}+(L\Delta)/T+\sigma^{2}G_{g}^{2}/s^{2}R^{2}) . Experiments on vision and language benchmarks demonstrate that DP-FedAdamW achieves a better privacy–utility trade-off and consistently outperforms SOTA DPFL baselines in convergence. 2 Related work Table 1: Comparison with existing DPFL optimizers. Method DP-FedAvg DP-SCAFFOLD DP-FedSAM DP-LocalAdamW DP-FedAdamW Optimizer Type SGD SGD SAM AdamW AdamW Scalability to Large Models Limited Limited Moderate Moderate High Client Drift Strong Weakened Slightly weakened Slightly weakened Weakened Communication Cost 1 \times 2 \times 2 \times 1 \times 1 \times Differentially Private Federated Learning (DPFL). We adopt sample-level DP for FL, ensuring privacy for each training example against any observer of the updates (the server or third parties). The most prevalent DP-FedAvg 52 applies per-sample clipping and Gaussian noise locally before aggregation. DP-SCAFFOLD 52 introduces control variates to alleviate client drift under privacy constraints. DP-FedAvg-LS 34 improves performance using Laplacian smoothing 53 . Beyond SGD, DP-FedSAM 62 uses Sharpness-Aware Minimization (SAM 21 ) to find flatter minima, improving robustness to DP noise. Despite progress in refined clipping and personalized DP 68 ; 44 , optimization remains the dominant bottleneck. However, these algorithms all employ the SGD optimizer for local updates and do not consider the use of adaptive optimizers. In the optimization of large-scale models, the AdamW optimizer has been shown to outperform SGD significantly in both convergence speed and final performance 76 ; 77 ; 75 ; 35 ; 8 ; 9 ; 4 ; 3 ; 5 ; 45 ; 46 ; 17 ; 14 ; 19 ; 18 ; 15 ; 66 ; 20 ; 13 ; 16 ; 36 ; 60 ; 61 ; 64 ; 63 ; 33 ; 31 ; 32 ; 50 ; 55 ; 56 ; 54 . Adaptive optimization. Adaptive optimizers such as Adam 27 , AdamW 48 , AMSGrad 59 , Amsgrad 58 adapt per-parameter step sizes from gradient history, achieving faster and more stable convergence than vanilla SGD and becoming standard for large models (e.g., Transformers) 74 ; 29 . Among these, AdamW applies weight decay directly on parameters, preventing undesired interaction between regularization and moment adaptation, thus improving convergence and generalization 48 ; 70 ; 39 ; 38 ; 37 . Adaptive optimization in Federated Learning. A line of work brings adaptive updates into FL. FedOpt 57 introduces server-side adaptivity, instantiating Adam and Yogi at the aggregator. FAFED 69 stabilizes training by aggregating clients’ first- and second-moment estimates of Adam. FedAMS 6 highlights that averaging the second-moment is pivotal to avoid divergence. FedLADA 65 aggregates the second-moment estimate to reduce communication cost. However, these studies mainly evaluate on CNNs. However, these algorithms do not take into account the impact of noise introduced by DP. FedBCGD 38 proposes an accelerated block coordinate gradient descent framework for FL. FedSWA 37 improves generalization under highly heterogeneous data by stochastic weight averaging. FedAdamW 42 introduces a communication-efficient AdamW-style optimizer tailored for federated large models. FedNSAM 40 studies the consistency relationship between local and global flatness in FL. FedMuon 41 accelerates federated optimization via matrix orthogonalization. DP-FedPGN 43 develops a penalizes gradient norms to encourage globally flatter minima in DP-FL. Our contributions. To the best of our knowledge, adaptive optimization has not yet been systematically explored in the context of DPFL. Existing DPFL optimizers still struggle to train large models in a stable and efficient manner. To address this gap, we propose a method that stabilizes second-moment statistics via block-wise aggregation, removes DP-induced bias in the second moment, and aligns local updates with an estimated global descent direction in a communication-efficient way. We summarize representative DPFL methods and their characteristics in Table 1 . 3 DPFL framework Consider a general DPFL system with N clients. The FL objective is to minimize the following population risk: f(\bm{\theta})=\frac{1}{N}\!\sum\nolimits_{i=1}^{N}f_{i}(\bm{\theta}),f_{i}(\bm{\theta}):=\mathbb{E}_{\xi_{i}\sim\mathcal{P}_{i}}\big[F_{i}(\bm{\theta};\xi_{i})\big], (1) where f_{i} is the loss function of client i . \xi_{i} is drawn from distribution \mathcal{P}_{i} in client i , and \mathbb{E}[\cdot] denotes expectation with respect to the sample \xi_{i} . In communication round t , the server selects S clients and distributes the global model \bm{\theta}^{t} . The client i computes the gradient for each sample j in local k iterations (with a mini-batch \mathcal{D}_{i}^{k}{=}\lfloor sR\rfloor , s is data sampling rate, R is number of client samples): g_{ij}=\nabla f_{i}(\bm{\theta}_{i}^{t,k};\xi_{ij}) , and performs gradient clipping and Gaussian noise addition to satisfy DP: \bar{g}_{ij}=g_{ij}/\max(1,\|g_{ij}\|_{2}/C), (2) \tilde{g}_{i}^{t,k}\leftarrow\frac{1}{sR}\sum_{j\in\mathcal{D}_{i}^{k}}\bar{g}_{ij}+\frac{C}{sR}\mathcal{N}\!\left(0,\sigma^{2}C^{2}I\right), (3) where C is the clipping threshold, \sigma is the noise factor. After K local steps, the server updates the global model \bm{\theta}^{t+1} by adding the averaged aggregate local update to \bm{\theta}^{t} . (a) Variance evolution of v (b) Client drift comparison Figure 3 : Training on CIFAR-100, Swin-Tiny, \sigma{=}1 , \alpha{=}0.1 . (a) Non-IID (DPFL) causes high variance in second-moment estimator across clients of DP-LocalAdamW. (b) DP-LocalAdamW suffers from more severe client drift than FedAvg and LocalAdamW. 4 Motivation and challenges Prior work 48 ; 78 shows that AdamW can substantially improve convergence and generalization. However, our key observation is that, in DPFL, AdamW fails to exhibit its typical advantages. In fact, in transformer-based models, AdamW sometimes performs worse than SGD, indicating the benefits of AdamW are diminished when privacy constraints and the FL setup introduce additional complexities. Challenge 1: Variance amplification of the second-moment estimator under non-IID and DP . In FL, non-IID data disperses client gradients, while DP noise further increases the variance of noisy gradients. This high variance accumulates in AdamW’s exponentially weighted moving averages of the first-moment \bm{m} and second-moment \bm{v} : \displaystyle\bm{m}_{i}^{t,k} \displaystyle\leftarrow\beta_{1}\bm{m}_{i}^{t,k-1}+(1-\beta_{1})\,\tilde{g}_{i}^{t,k}, (4) \displaystyle\bm{v}_{i}^{t,k} \displaystyle\leftarrow\beta_{2}\bm{v}_{i}^{t,k-1}+(1-\beta_{2})\,\tilde{g}_{i}^{t,k}\odot\tilde{g}_{i}^{t,k}, where \beta_{1}{=}0.9 , \beta_{2}{=}0.999 are the exponential decay rates. \odot represents the elementwise (Hadamard) product. \displaystyle\operatorname{Var}(v^{t}) \displaystyle\approx\frac{(1-\beta_{2})^{2}}{1-\beta_{2}^{2}}\,\operatorname{Var}\!\left(\tilde{g}^{t}\odot\tilde{g}^{t}\right). (5) Since \tilde{g}^{t}\odot\tilde{g}^{t} magnifies noise and \beta_{2}\!\approx\!1 induces slow averaging, the variance of \bm{v} grows rapidly and dominates the optimization noise under DP and heterogeneity. Figure 3 (a) shows the variance of \bm{v} under non-IID+DP remains higher and converges much slower than in IID or non-DP cases. (a) Histogram of \bm{m}^{t} (b) Histogram of \sqrt{\bm{v}^{t}} Figure 4 : Histogram for DP-LocalAdamW, CIFAR-10, Swin-Tiny, \sigma{=}1 , \alpha{=}0.1 . (a) The distribution centers of \bm{m}^{t} are aligned with or without DP, but the variance is larger with DP. (b) The distribution of \sqrt{\bm{v}^{t}} shows a significant difference, with the center of the distribution shifting approximately by \sqrt{\sigma^{2}C^{2}/(sR)^{2}} . Challenge 2: Systematic bias in the second-moment estimator introduced by DP. Under DP clipping and noise, directly using the perturbed gradient to estimate the second moment produces a systematic bias. This bias is additive and distinct from the initialization bias addressed by Adam’s exponential moving average, so that correction does not remove it in Figure 4 . Recall the definition of \bm{v}_{i}^{t,k} from Eq. 4 , Eq. 2 , Eq. 3 , we have \displaystyle\mathbb{E}\!\left[\tilde{g}_{i}^{t,k}\odot\tilde{g}_{i}^{t,k}\right]=\mathbb{E}\!\left[\bar{g}_{i}^{t,k}\odot\bar{g}_{i}^{t,k}\right]+{\sigma^{2}C^{2}}/{(sR)^{2}}\,I, (6) \displaystyle\mathbb{E}\!\left[v_{i}^{t,k}\right]=\mathbb{E}\!\left[v_{i}^{t,k}\right]_{\mathrm{w/o\ DP}}+\bigl(1-\beta_{2}^{\,k}\bigr)\,{\sigma^{2}C^{2}}/{(sR)^{2}}\,I. Challenge 3: Client drift caused by local adaptivity and DP mechanism. In FL, each client optimizes its local objective f(\cdot) , creating a gap between local and global optima. AdamW accelerates this by pushing clients toward their local optima, increasing client drift. In DPFL, gradient clipping and noise injection reduce effective sample sizes, making updates noisier and amplifying overfitting. This exacerbates client drift, hindering global model convergence. Figure 3 (b) shows that DP-FedAdamW results in higher client drift compared to FedAvg and LocalAdamW, where LocalAdamW refers to Federated AdamW without DP. Algorithm 1 DP-FedAdamW 1: Initial initial model \bm{\theta}^{0} ; rounds T ; local steps K ; step size \eta ; weight decay \lambda ; AdamW (\beta_{1},\beta_{2},\epsilon) ; clip norm C ; noise multiplier \sigma ; S clients per round 2: for t=1 to T do 3: for selected clients i=1,\dots,S in parallel do 4: \bm{\theta}_{i}^{t,0}\leftarrow\bm{\theta}^{t} ; \bm{m}_{i}^{t,0}\leftarrow\bm{0} ; \bm{v}_{i}^{t,0}\leftarrow\bm{\bar{v}}^{t} 5: for k=1 to K do 6: sample \mathcal{D}_{i}^{k}\subset\mathcal{D}_{i} of size \lfloor sR\rfloor 7: for each sample j\in\mathcal{D}_{i}^{k} do 8: g_{ij}\leftarrow\nabla f_{i}(\bm{\theta}_{i}^{t,k};\xi_{ij}) 9: \bar{g}_{ij}\leftarrow g_{ij}/\max(1,\|g_{ij}\|_{2}/C) 10: end for 11: \tilde{g}_{i}^{t,k}\leftarrow\frac{1}{sR}\sum_{j\in\mathcal{D}_{i}^{k}}\bar{g}_{ij}+\frac{C}{sR}\mathcal{N}\!\left(0,\sigma^{2}C^{2}I\right) 12: \bm{m}_{i}^{t,k}\leftarrow\beta_{1}\bm{m}_{i}^{t,k-1}+(1-\beta_{1})\,\tilde{g}_{i}^{t,k} 13: \bm{v}_{i}^{t,k}\leftarrow\beta_{2}\bm{v}_{i}^{t,k-1}+(1-\beta_{2})\,\tilde{g}_{i}^{t,k}\odot\tilde{g}_{i}^{t,k} 14: \hat{\bm{m}}_{i}^{t,k}\leftarrow\bm{m}_{i}^{t,k}/(1-\beta_{1}^{k}) 15: \hat{\bm{v}}_{i}^{t,k}\leftarrow\bm{v}_{i}^{t,k}/(1-\beta_{2}^{k}) 16: \vartheta_{i}^{t,k}\leftarrow 1/(\sqrt{\hat{\bm{v}}_{i}^{t,k}-\left(\frac{\sigma C}{sR}\right)^{2}}+\epsilon) 17: \bm{\theta}_{i}^{t,k+1}\!\leftarrow\!\bm{\theta}_{i}^{t,k}\!-\!\eta(\hat{\bm{m}}_{i}^{t,k}\!\odot\!\vartheta_{i}^{t,k}\!+\!\gamma\bm{\Delta}_{G}^{t}\!-\!\lambda\bm{\theta}_{i}^{t,k}) 18: end for 19: Send (\bm{\theta}^{t,K}_{i}\!\!-\!\!\bm{\theta}^{t,0}_{i},\bm{\bar{v}}_{i}\!=\!\texttt{Block\_mean}(\bm{v}^{t,K}_{i})) 20: end for 21: \bm{\Delta}_{G}^{t+1}=\frac{-1}{SK\eta}\sum_{i=1}^{S}(\bm{\theta}^{t,K}_{i}-\bm{\theta}^{t,0}_{i}) 22: \bm{\theta}^{t+1}=\bm{\theta}^{t}+\frac{1}{S}\sum_{i=1}^{S}(\bm{\theta}^{t,K}_{i}-\bm{\theta}^{t,0}_{i}) 23: \bm{\bar{v}}^{t+1}=\frac{1}{S}\sum_{i=1}^{S}\bm{\bar{v}}_{i} 24: Broadcast (\bm{\theta}^{t+1},\bm{\bar{v}}^{t+1},\bm{\Delta}_{G}^{t+1}) to clients 25: end for 5 Methodology To restore the effectiveness of AdamW in DPFL, we propose Differentially Private Federated AdamW (DP-FedAdamW). DP-FedAdamW is built upon a unified perspective: the moment statistics of AdamW become distorted under non-IID data and DP. Our design corrects this distortion along three complementary axes: parameter-block aggregation to stabilize second-moment variance, explicit removal of DP-induced bias in the second moment, and local–global update alignment to reduce client drift. 5.1 Second-moment aggregation To address Challenge 1 , we stabilize the training process by aggregating the second-moment estimates of AdamW. However, directly aggregating per-parameter second moments requires communicating an additional vector with the same dimensionality as the model parameters, which almost doubles the communication overhead. We therefore introduce a block-wise aggregation scheme that communicates only one statistic per parameter block. We adopt a parameter-block averaging strategy, where each block shares a single learning rate to capture local curvature effectively. Based on this idea, we partition the parameters into B blocks and compute the mean of second-moments within each block (line 19 in Algorithm 1 ): \bm{\bar{v}}_{b}=\frac{1}{|\bm{v}_{b}|}\sum_{i\in\bm{v}_{b}}v_{i},\quad b=1,\dots,B. (7) Block-wise averaging strategy. We design the blocks to align with the model architecture. The attention-related parameters are partitioned at the granularity of attention heads, while the remaining modules are partitioned at the granularity of whole layers. For each parameter block, we compute the mean of its second-moment estimates and transmit only this block-wise statistic: • Head-wise Q , K , V . According to the number of attention heads in the model, each head-specific slice of the query ( Q ), key ( K ), and value ( V ) weight matrix is treated as an independent parameter block. For every such Q/K/V block, we compute and transmit the mean of its second-moment estimates. • attn.proj and MLP layers. Each attention output projection layer ( attn.proj ) and each MLP layer is regarded as a single parameter block. For each layer, we aggregate the second-moment estimates of all parameters within the layer and transmit the block-wise mean . • Embedding and output layers. Similarly, each Embedding layer and each output (e.g., classification) layer is treated as one parameter block. For these blocks, we also compute and transmit the mean of the corresponding second-moment estimates. CNNs (e.g., ResNet): For convolutional networks, we define each convolutional layer or residual block as one parameter block. This reduces the communication cost from per-parameter second moments to B block-level values while preserving adaptive behavior across different components of the network. 5.2 Unbiased second-moment correction To address Challenge 2 , we propose a Bias-Corrected (BC). The correction term subtracts the variance contribution of the noise from the second-moment estimate of DP-AdamW, restoring the scaling behavior of non-private Adam and removing the influence of DP on the second-moment estimation. The update rule is given by (line 16 in Algorithm 1 ): \vartheta_{i}^{t,k}=1/(\sqrt{\hat{\bm{v}}_{i}^{t,k}-\left(\frac{\sigma C}{sR}\right)^{2}}+\epsilon) (8) Here, \hat{\bm{v}}_{i}^{t,k} denotes the exponential moving average of the (noisy) second moment of the gradient for parameter i , \sigma is the noise multiplier, C is the clipping norm, and sR is the batch size. The term \frac{\sigma C}{sR} corresponds to the variance of the Gaussian noise added for differential privacy, and subtracting it removes the constant bias introduced by DP noise in the second-moment estimate. 5.3 Local-global alignment To mitigate Challenge 3 (client drift under non-IID data with DP), we augment the local AdamW update with an explicit alignment toward the global descent direction. Concretely, we modify the local update rule (line 17 in Algorithm 1 ) as \bm{\theta}_{i}^{t,k+1}=\bm{\theta}_{i}^{t,k}-\eta\left(\hat{\bm{m}}^{t,k}_{i}\odot\vartheta_{i}^{t,k}-\lambda\bm{\theta}_{i}^{t,k}+\gamma\bm{\Delta}_{G}^{t}\right), (9) where \bm{\Delta}_{G}^{t}=\frac{-1}{SK\eta}\sum_{i=1}^{S}(\bm{\theta}_{i}^{t,K}\!-\!\bm{\theta}_{i}^{t,0}) is an empirical estimate of the global update over S participating clients. Under non-IID data, the adaptive local gradient \hat{\bm{m}}^{t,k}_{i}\odot\vartheta_{i}^{t,k} is biased toward each client’s own objective. DP clipping distorts the magnitude and direction of these gradients, and DP noise adds high-variance perturbations, pushing the local parameters \bm{\theta}_{i}^{t,k} toward different client-specific optima and amplifying the client drift identified in Challenge 3 . The alignment term \gamma\bm{\Delta}_{G}^{t} softly regularizes each local step toward the global descent direction. It is nearly inactive when a client’s update already follows the global trend, but when non-IID data and DP operations drive the local direction away, the alignment term counteracts this deviation and steers the trajectory back toward the global path. As illustrated in Figure 2 , this local–global alignment tightens the spread of client models, reduces cross-client update variance at aggregation, and yields a smoother, more stable global optimization trajectory under strong non-IID and DP. 6 Theoretical analysis 6.1 Convergence analysis In this part, we give the convergence theoretical analysis of our proposed DP-FedAdamW algorithm. Firstly, we give some standard assumptions for the non-convex function f . Assumption 1 (Smoothness) . (Smoothness) The non-convex f_{i} is a L -smooth function for all i\in[m] , i.e., \|\nabla f_{i}(\bm{\theta}_{1})-\nabla f_{i}(\bm{\theta}_{2})\|\leq L\|\bm{\theta}_{1}-\bm{\theta}_{2}\| , for all \bm{\theta}_{1},\bm{\theta}_{2}\in\mathbb{R}^{d} . Assumption 2 (Bounded Stochastic Gradient) . \bm{g}_{i}^{t}=\nabla f_{i}(\bm{\theta}_{i}^{t},\xi_{i}^{t}) computed by using a sampled mini-batch data \xi_{i}^{t} in the local client i is an unbiased estimator of \nabla f_{i} with bounded variance, i.e., \mathbb{E}_{\xi_{i}^{t}}[\bm{g}_{i}^{t}]=\nabla f_{i}(\bm{\theta}_{i}^{t}) and \mathbb{E}_{\xi_{i}^{t}}\|g_{i}^{t}-\nabla f_{i}(\bm{\theta}_{i}^{t})\|^{2}\leq\sigma_{l}^{2} , for all \bm{\theta}_{i}^{t}\in\mathbb{R}^{d} . Assumption 3 (Bounded Stochastic Gradient II) . Each element of stochastic gradient \bm{g}_{i}^{t} is bounded, i.e., \|\bm{g}_{i}^{t}\|_{\infty}=\|f_{i}(\bm{\theta}_{i}^{t},\xi_{i}^{t})\|_{\infty}\leq G_{g} , for all \bm{\theta}_{i}^{t}\in\mathbb{R}^{d} and any sampled mini-batch data \xi_{i}^{t} . Theorem 1 (Convergence for non-convex functions) . Under Assumptions A.1 , A.2 , and A.3 , if we take g^{0}=0 , then DP-FedAdamW converges as follows \frac{1}{T}\!\sum_{t=0}^{T-1}\!\mathbb{E}\!\left[\|\nabla f(\bm{\theta}^{t})\|^{2}\right]\lesssim\mathcal{O}\!\left(\sqrt{\frac{L\Delta\sigma_{l}^{2}}{SKT\epsilon^{2}}}+\frac{L\Delta}{T}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}R^{2}}\right). (10) Here G_{0}:=\frac{1}{N}\sum_{i=1}^{N}\left\|\nabla f_{i}\left(\bm{\theta}^{0}\right)\right\|^{2} , \Delta=f\left(\bm{\theta}^{0}\right)-f^{\star} , S is the number of participating clients per round, \sigma is DP noise level, \sigma_{l} is noise of stochastic gradient, K is the number of local iterations, and T is the total number of communication rounds. The proof is provided in Appendix 11.1 . The convergence rate of DP-FedAdamW is faster than that of DP-LocalAdamW’s \mathcal{O}\left(\sqrt{\frac{L\Delta(\sigma_{l}^{2}+\sigma_{g}^{2})}{SKT\epsilon^{2}}}+\frac{L\Delta}{T}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}R^{2}}\right) , and we do not need bounded heterogeneity assumption. This is due to the suppression of local drift by the global update estimation \bm{\Delta}_{G}^{t} . Unlike prior adaptive FL optimizers 65 which rely on heterogeneity assumption, our convergence result holds without such restrictions, making it more widely applicable. 6.2 Privacy analysis Differential privacy (DP) 10 provides a formal framework for quantifying disclosure risk. We conduct privacy accounting under Rényi DP (RDP) 51 , which gives tight bounds under composition and subsampling. Following 52 , we finally convert RDP to (\varepsilon,\delta) -DP 51 . Definition 1 . (Sample-level DP, 22 ). For any two neighboring datasets \mathcal{D} and \mathcal{D}^{\prime} that differ in a single sample or record (either by addition or removal), a randomized mechanism \mathcal{M}:\mathcal{X}^{n}\!\to\!\mathcal{Y} satisfies (\varepsilon,\delta) -DP at the sample level if, for every possible output set O\subseteq\mathbb{Y} , the following holds: \operatorname{Pr}[\mathcal{M}(\mathcal{D})\in O]\leq e^{\varepsilon}\operatorname{Pr}\left[\mathcal{M}\left(\mathcal{D}^{\prime}\right)\in O\right]+\delta, (11) where \varepsilon is the privacy budget, \delta is the failure probability. Definition 2 . (Rényi DP, 51 ). For any order \zeta\in(1,\infty) , \mathcal{M} satisfies (\zeta,\varepsilon_{\mathrm{T}}) -RDP if for any two neighboring datasets \mathcal{D} , \mathcal{D}^{\prime} that differ by a single sample, it has: \!\!\!D_{\zeta}\left[\mathcal{M}(\mathcal{D})\|\mathcal{M}\left(\mathcal{D}^{\prime}\right)\right]\!:=\!\frac{1}{\zeta\!-\!1}\log\mathbb{E}\left[\left(\frac{p_{\mathcal{M}(\mathcal{D})}\left(Y\right)}{p_{\mathcal{M}\left(\mathcal{D}^{\prime}\right)}\left(Y\right)}\right)^{\zeta}\!\right]\!\leq\!\varepsilon_{\mathrm{R}}, (12) The expectation \mathbb{E} is taken over the output Y\sim\mathcal{M}(\mathcal{D}^{\prime}) . We adopt standard RDP accounting following 52 , the cumulative privacy over K local steps and T rounds is: Theorem 2 (Privacy guarantee) . For DP-FedAdamW, \bm{\theta}^{T} is (\varepsilon,\delta) -DP towards a third party \varepsilon=\mathcal{O}\left(\frac{s\sqrt{TK\log(2/\delta)\log(2T/\delta)}}{\sigma}\right),\vskip-2.84526pt (13) Towards the server, accumulative (\varepsilon_{s},\delta_{s}) -DP is \varepsilon_{s}=\varepsilon\sqrt{\frac{N}{l}},\delta_{s}=\frac{\delta}{2}\left(\frac{1}{l}+1\right), (14) where N is the number of clients, l is clients sample rate, and s is client data sample rate. Table 2: Averaged test accuracy (%) and privacy budget comparison on CIFAR-10 and CIFAR-100 using ResNet-18 and ViT-Base. Assuming DP-FedAvg is (\varepsilon,\delta) -DP, all other methods share this budget except DP-FedSAM, which is approximately (2\varepsilon,\delta) -DP. Method CIFAR-10 (ResNet-18) CIFAR-100 (ResNet-18) CIFAR-10 (ViT-Base) CIFAR-100 (ViT-Base) Privacy Budget \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 DP-FedAvg 71.69_{\pm 0.37} 60.38_{\pm 0.21} 31.13_{\pm 0.57} 25.04_{\pm 0.72} 90.51_{\pm 0.37} 78.85_{\pm 0.21} 45.86_{\pm 0.57} 13.41_{\pm 0.72} (\varepsilon,\delta) DP-SCAFFOLD 71.79_{\pm 0.44} 59.04_{\pm 0.39} 31.52_{\pm 0.36} 24.94_{\pm 0.58} 90.88_{\pm 0.52} 80.69_{\pm 0.64} 62.45_{\pm 0.57} 64.69_{\pm 0.76} (\varepsilon,\delta) DP-FedAvg-LS 72.27_{\pm 0.57} 60.84_{\pm 0.45} 35.91_{\pm 0.54} 28.68_{\pm 0.52} 91.48_{\pm 0.47} 88.48_{\pm 0.50} 67.76_{\pm 0.58} 65.85_{\pm 0.63} (\varepsilon,\delta) DP-FedSAM 67.39_{\pm 0.41} 57.04_{\pm 0.46} 30.36_{\pm 0.50} 24.03_{\pm 0.66} 91.76_{\pm 0.55} 89.35_{\pm 0.65} 70.24_{\pm 0.63} 66.27_{\pm 0.65} (2\varepsilon,\delta) DP-LocalAdamW 71.73_{\pm 0.47} 60.55_{\pm 0.53} 34.92_{\pm 0.69} 28.70_{\pm 0.81} 91.16_{\pm 0.47} 89.85_{\pm 0.53} 68.21_{\pm 0.69} 65.24_{\pm 0.81} (\varepsilon,\delta) DP-FedAdamW \color[rgb]{1,0,0}{74.81}_{\pm 0.19} \color[rgb]{1,0,0}{64.65}_{\pm 0.24} \color[rgb]{1,0,0}{39.56}_{\pm 0.42} \color[rgb]{1,0,0}{33.55}_{\pm 0.50} \color[rgb]{1,0,0}{92.02}_{\pm 0.49} \color[rgb]{1,0,0}{90.30}_{\pm 0.24} \color[rgb]{1,0,0}{71.59}_{\pm 0.42} \color[rgb]{1,0,0}{67.56}_{\pm 0.50} (\varepsilon,\delta) Table 3: Averaged test accuracy (%) and privacy budget comparison on CIFAR-100 and Tiny-ImageNet using Swin-Tiny and Swin-Base. Method CIFAR-100 (Swin-T) CIFAR-100 (Swin-B) Tiny-ImageNet (Swin-T) Tiny-ImageNet (Swin-B) Privacy Budget \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 DP-FedAvg 36.96_{\pm 0.50} 28.07_{\pm 0.31} 65.61_{\pm 0.27} 60.77_{\pm 0.32} 18.82_{\pm 0.94} 10.12_{\pm 1.36} 52.04_{\pm 0.66} 36.19_{\pm 0.74} (\varepsilon,\delta) DP-SCAFFOLD 41.68_{\pm 0.64} 33.52_{\pm 0.65} 68.03_{\pm 0.55} 65.34_{\pm 0.49} 21.91_{\pm 0.78} 14.54_{\pm 0.72} 53.78_{\pm 0.79} 43.55_{\pm 0.60} (\varepsilon,\delta) DP-FedAvg-LS 41.95_{\pm 0.53} 36.72_{\pm 0.65} 68.80_{\pm 0.52} 66.08_{\pm 0.56} 23.03_{\pm 0.64} 15.99_{\pm 0.68} 55.83_{\pm 0.66} 47.46_{\pm 0.57} (\varepsilon,\delta) DP-FedSAM 44.19_{\pm 0.48} 38.43_{\pm 0.62} 70.44_{\pm 0.57} 66.28_{\pm 0.64} 22.75_{\pm 0.65} 16.86_{\pm 0.77} 56.12_{\pm 0.74} 48.08_{\pm 0.62} (2\varepsilon,\delta) DP-LocalAdamW 42.04_{\pm 0.63} 35.51_{\pm 0.58} 68.33_{\pm 0.33} 65.28_{\pm 0.39} 23.41_{\pm 0.87} 15.81_{\pm 0.73} 54.58_{\pm 0.60} 46.52_{\pm 0.64} (\varepsilon,\delta) DP-FedAdamW \color[rgb]{1,0,0}{46.77}_{\pm 0.38} \color[rgb]{1,0,0}{39.36}_{\pm 0.55} \color[rgb]{1,0,0}{71.58}_{\pm 0.30} \color[rgb]{1,0,0}{67.45}_{\pm 0.26} \color[rgb]{1,0,0}{23.96}_{\pm 0.50} \color[rgb]{1,0,0}{19.19}_{\pm 0.55} \color[rgb]{1,0,0}{58.44}_{\pm 0.54} \color[rgb]{1,0,0}{50.76}_{\pm 0.51} (\varepsilon,\delta) Table 4: Averaged test accuracy (%) on RoBERTa-Base. Method SST-2 QQP QNLI MNLI DP-FedAvg 49.08_{\pm 0.12} 63.18_{\pm 0.50} 49.46_{\pm 0.44} 35.45_{\pm 0.20} DP-SCAFFOLD 52.35_{\pm 0.18} 66.87_{\pm 0.42} 52.62_{\pm 0.52} 38.74_{\pm 0.32} DP-FedAvg-LS 68.38_{\pm 0.50} 72.59_{\pm 0.48} 69.22_{\pm 0.68} 50.46_{\pm 0.55} DP-FedSAM 70.95_{\pm 0.62} 75.66_{\pm 0.53} 70.54_{\pm 0.63} 56.88_{\pm 0.56} DP-LocalAdamW 91.04_{\pm 0.53} 82.33_{\pm 0.59} 86.07_{\pm 0.47} 75.20_{\pm 0.44} DP-FedAdamW \color[rgb]{1,0,0}{91.17}_{\pm 0.32} \color[rgb]{1,0,0}{83.34}_{\pm 0.39} \color[rgb]{1,0,0}{86.33}_{\pm 0.34} \color[rgb]{1,0,0}{78.68}_{\pm 0.38} Table 5: Effect of privacy budgets ( \varepsilon ) on test accuracy (%) using Swin-Base, \alpha=0.1 . Method CIFAR-10 CIFAR-100 Tiny-ImageNet \varepsilon{=}1 \varepsilon{=}2 \varepsilon{=}3 \varepsilon{=}1 \varepsilon{=}2 \varepsilon{=}3 \varepsilon{=}1 \varepsilon{=}2 \varepsilon{=}3 DP-FedAvg 66.52_{\pm 0.89} 72.63_{\pm 0.55} 88.02_{\pm 0.76} 19.66_{\pm 1.52} 42.98_{\pm 0.60} 57.73_{\pm 0.55} 20.65_{\pm 0.77} 37.48_{\pm 0.54} 45.32_{\pm 0.47} DP-SCAFFOLD 66.39_{\pm 0.68} 72.81_{\pm 0.53} 88.07_{\pm 0.66} 22.38_{\pm 0.67} 44.52_{\pm 0.68} 59.85_{\pm 0.52} 20.93_{\pm 0.66} 38.40_{\pm 0.51} 47.06_{\pm 0.30} DP-FedAvg-LS 71.57_{\pm 1.02} 75.02_{\pm 0.59} 88.96_{\pm 0.54} 25.77_{\pm 0.61} 49.54_{\pm 0.59} 62.46_{\pm 0.53} 26.89_{\pm 0.46} 46.57_{\pm 0.44} 48.83_{\pm 0.21} DP-FedSAM 53.51_{\pm 1.39} 59.33_{\pm 0.86} 76.82_{\pm 0.97} 12.33_{\pm 0.64} 28.32_{\pm 0.49} 46.93_{\pm 0.50} 10.88_{\pm 0.95} 26.76_{\pm 0.78} 41.69_{\pm 0.45} DP-LocalAdamW 66.57_{\pm 1.35} 73.04_{\pm 0.58} 88.83_{\pm 0.71} 24.69_{\pm 0.88} 47.26_{\pm 0.59} 62.18_{\pm 0.61} 28.40_{\pm 0.72} 46.62_{\pm 0.37} 48.55_{\pm 0.28} DP-FedAdamW \color[rgb]{1,0,0}{77.50}_{\pm 0.54} \color[rgb]{1,0,0}{80.37}_{\pm 0.58} \color[rgb]{1,0,0}{89.98}_{\pm 0.55} \color[rgb]{1,0,0}{30.06}_{\pm 0.41} \color[rgb]{1,0,0}{52.11}_{\pm 0.54} \color[rgb]{1,0,0}{65.47}_{\pm 0.42} \color[rgb]{1,0,0}{34.23}_{\pm 0.50} \color[rgb]{1,0,0}{50.95}_{\pm 0.26} \color[rgb]{1,0,0}{52.53}_{\pm 0.22} (a) ResNet-18, \alpha=0.6 (b) ResNet-18, \alpha=0.1 (c) Swin-Tiny, \alpha=0.6 (d) Swin-Tiny, \alpha=0.1 Figure 5 : Test accuracy (%) on CIFAR-100 using ResNet-18 and Swin-Tiny under the Dirichlet \alpha=0.6 and \alpha=0.1 settings. Table 6: Ablation on the main components of DP-FedAdamW on Swin-Base. We report test accuracy (%) on CIFAR-100 and Tiny-ImageNet under non-IID setting ( \alpha=0.1 ). “Agg” denotes block-wise second-moment aggregation, “BC” the DP bias correction, and “Align” the local–global alignment term. Method CIFAR-100 Tiny-ImageNet DP-LocalAdamW 65.28 46.52 w/o Agg 66.41 47.35 w/o BC 67.02 48.11 w/o Align 66.37 47.82 DP-FedAdamW (full) \color[rgb]{1,0,0}{67.42} \color[rgb]{1,0,0}{50.73} Table 7: Ablation study of moment aggregation strategies of DP-LocalAdamW on CIFAR-100 with Swin-Base under \alpha{=}0.1 . Strategy CIFAR-100 Tiny-ImageNet Comm(↑) NoAgg 66.21_{\pm 0.11} 49.12_{\pm 0.12} 5.7 M Agg- \bm{v} 67.52_{\pm 0.12} 50.88_{\pm 0.12} 11.4 M Agg- mean - \bm{v} \color[rgb]{1,0,0}{67.45}_{\pm 0.10} \color[rgb]{1,0,0}{50.76}_{\pm 0.10} 5.7M Table 8: Impact of \gamma using Swin-Base ( \alpha{=}0.1 ). \gamma 0.00 0.25 \color[rgb]{1,0,0}0.50 0.75 1.00 CIFAR-100 (%) 66.37 67.01 \color[rgb]{1,0,0}67.45 67.10 66.52 Tiny-ImageNet (%) 47.82 49.36 \color[rgb]{1,0,0}50.76 50.21 49.18 Table 9: Comparison with DP variants of federated adaptive optimizers under the same DP settings (Swin-Base, \alpha{=}0.1 , \sigma{=}1 ). Method CIFAR-100 Tiny-ImageNet DP-FedOpt(Adam) 57 {64.54}_{\pm 0.56} {44.81}_{\pm 0.62} DP-FAFED 69 65.75_{\pm 0.54} 46.10_{\pm 0.67} DP-FedLADA 65 66.03_{\pm 0.61} 47.85_{\pm 0.57} DP-FedAdamW (ours) \color[rgb]{1,0,0}{67.45}_{\pm 0.26} \color[rgb]{1,0,0}{50.76}_{\pm 0.51} 7 Experiments 7.1 Experimental setup DPFL Algorithms. We conduct experiments on six DPFL algorithms: DP-FedAvg 52 , DP-SCAFFOLD 52 , DP-FedAvg-LS 34 , DP-FedSAM 62 , DP-LocalAdamW, and our DP-FedAdamW. In all performance evaluation experiments, the DP noise multiplier \sigma is fixed. Under the same clipping norm and communication rounds, DP-FedSAM requires roughly twice the privacy budget of other methods, as it injects independent DP noise into two gradient evaluations at each local step. All methods are implemented using PyTorch and executed on NVIDIA RTX 4090 GPUs. Datasets and data partition. We evaluate DP-FedAdamW on seven datasets covering both vision and language tasks. For image classification, we adopt CIFAR-10, CIFAR-100 28 , and Tiny-ImageNet 30 . For natural language understanding, we consider four GLUE 67 benchmarks: SST-2, QQP, QNLI, and MNLI. We simulate non-IID client data using a Dirichlet partition 25 , where each client’s distribution is sampled from Dir( \alpha ) with \alpha\in\{0.1,0.6,0.8\} . Model architectures. We study three representative architecture families. (1) convolutional neural networks(CNNs): GNResNet-18 evaluated on CIFAR-10/100. (2) vision transformers: ViT-Base for CIFAR-10/100 and Swin-Tiny/Base for CIFAR-10/100 and Tiny-ImageNet. (3) language transformer: RoBERTa-Base assessed on the GLUE. Configurations. For DP-FedAvg, DP-SCAFFOLD, DP-FedAvg-LS, and DP-FedSAM, the learning rate \eta is selected from \{10^{-2},\,3\times 10^{-2},\,5\times 10^{-2},\,10^{-1},\,3\times 10^{-1}\} with a weight decay of 0.001 . For DP-LocalAdamW, DP-FedAdamW, \eta is selected from \{10^{-4},\,3\times 10^{-4},\,5\times 10^{-4},\,8\times 10^{-4},\,10^{-3}\} with weight decay 0.01 or 0.001 , \beta_{1}=0.9 , \beta_{2}=0.999 . We apply cosine learning rate decay, and set DP-FedAdamW to \gamma{=}0.5 , weight decay \lambda{=}0.01 . For ResNet-18, ronds T{=}300 , batch size 50 , local step K{=}50 . For Transformers, ronds T{=}100 , batch size 16 , local step K{=}20 . The noise multiplier is \sigma{=}1 . The privacy failure probability is set to \delta=10^{-5} for CIFAR10/100 and \delta=10^{-6} for Tiny-ImageNet. Please refer to Appendix 9.4 for more details of the experimental settings. All results are averaged over 5 runs with std reported. 7.2 Performance on vision datasets ResNet-18 on CIFAR-10/100. Table 2 summarizes the results on CIFAR-10/100 under different heterogeneity levels ( \alpha{=}0.6 and \alpha{=}0.1 ). DP-FedAdamW achieves the best performance across all settings, reaching 74.81\% and 64.65\% on CIFAR-10, and 39.56\% and 33.55\% on CIFAR-100. When the data heterogeneity is severe ( \alpha{=}0.1 ), DP-FedAdamW surpasses the SOTA baselines (DP-FedAvg-LS for CIFAR-10 and DP-LocalAdamW for CIFAR-100) by \mathbf{3.81\%} and \mathbf{4.85\%} , respectively. ViT-Base on CIFAR-10/100. We fine-tune ViT-Base on CIFAR-10/100, and the results are reported in Table 2 under \alpha{=}0.6 and \alpha{=}0.1 . DP-FedAdamW consistently achieves the best performance. On CIFAR-100 with \alpha{=}0.1 , it outperforms the strongest baseline DP-FedSAM by \mathbf{1.29\%} . Recall that, under our DP configuration, DP-FedSAM is trained with a much weaker privacy guarantee (it uses twice the privacy budget of the other methods). Swin-Tiny/Base on CIFAR-10/100/Tiny-ImageNet. We fine-tune Swin-Tiny and Swin-Base on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and report the results in Tables 3 , 10 . DP-FedAdamW consistently attains the highest accuracy across all settings. For example, on CIFAR-100 with Swin-Tiny at \alpha{=}0.6 , it reaches 46.77\% , outperforming the strongest baseline DP-FedSAM by \mathbf{2.58\%} . on Tiny-ImageNet with Swin-Base at \alpha{=}0.1 , it achieves 50.85\% , exceeding DP-FedSAM by \mathbf{2.77\%} . Figure 5 (c,d) shows that DP-FedAdamW achieves advantage on Swin-Tiny. 7.3 Performance on language datasets RoBERTa-Base on GLUE. We further evaluate our method on four GLUE tasks (SST-2, QQP, QNLI, MNLI) by fine-tuning RoBERTa-Base under \alpha{=}0.8 , and report the results in Table 4 . DP-FedAdamW attains the best accuracy on all tasks, reaching 91.17\% , 83.34\% , 86.33\% , and 78.68\% , respectively. on the MNLI, DP-FedAdamW outperforms the baseline DP-LocalAdamW by \mathbf{3.48\%} , showing that the benefits of DP-FedAdamW extend from vision benchmarks to language understanding in the DPFL setting. 7.4 Discussion for DP with AdamW in FL Performance under different privacy budgets \varepsilon . Table 5 reports the results of Swin-Base on CIFAR-10, CIFAR-100, and Tiny-ImageNet under \varepsilon{=}1,2,3 with \alpha{=}0.1 . As expected, all methods benefit from larger \varepsilon (weaker privacy). Notably, the performance gap between DP-FedAdamW and other methods is most pronounced when \varepsilon is small, i.e., under stronger privacy constraints. For instance, on CIFAR-10, DP-FedAdamW achieves 77.50\% at \varepsilon{=}1 , surpassing the best baseline by \mathbf{5.93\%} . These results show that DP-FedAdamW achieves a superior utility–privacy trade-off. Noise multiplier \sigma . Results are provided in Appendix 9.1 . 7.5 Ablation study Effect of each component. DP-FedAdamW differs from DP-LocalAdamW in three aspects: (i) block-wise second-moment aggregation (Agg); (ii) DP bias correction on the second-moment (BC); (iii) local–global alignment term (Align). We construct several variants by removing one component at a time, while keeping all other settings fixed. Table 9 shows that each component yields a gain, and their combination achieves the best performance. Bias correction (BC) improves robustness to DP noise, the local–global alignment stabilizes optimization under non-IID and DP, and block-wise aggregation provides an additional improvement with negligible communication overhead. Impact of aggregation strategy. Table 7 shows that our block-wise averaging strategy, Agg- mean - \bm{v} , achieves the best balance between accuracy and communication cost. Agg- mean - \bm{v} achieves comparable gains with only \mathcal{O}(B) communication, where B is the number of blocks, demonstrating its scalability and ability to stabilize updates even under strict communication budgets. Effect of the alignment coefficient \gamma . Table 8 varies \gamma on Swin-Base ( \alpha{=}0.1 ). Increasing \gamma from 0.0 to 0.5 improves accuracy on CIFAR-100 and Tiny-ImageNet, indicating that a moderate alignment term effectively counteracts client drift and stabilizes optimization under DP noise. Further increasing \gamma beyond 0.5 leads to a slight degradation, as overly strong alignment over-constrains local updates. We therefore set \gamma{=}0.5 as the default. Comparison with DP variants of federated adaptive optimizers. As shown in Table 9 , on Swin-Base with Dirichlet \alpha{=}0.1 , DP noise \sigma{=}1 , simply adding DP to existing federated adaptive optimizers yields limited gains. DP-FAFED and DP-FedLADA slightly outperform DP-FedOpt(Adam) on both CIFAR-100 and Tiny-ImageNet. Our DP-FedAdamW consistently achieves the best performance, with roughly 1.42\% on CIFAR-100 and 2.91\% on Tiny-ImageNet compared to the SOTA DP baseline. 8 Conclusion In this work, we revisited AdamW for differentially private federated learning and revealed that a naïve adoption leads to amplified second-moment variance, systematic DP-induced bias, and pronounced client drift under small-batch local training. Building on these insights, we introduced DP-FedAdamW, which integrates block-wise second-moment aggregation, explicit DP bias correction, and a local–global alignment mechanism, with non-convex convergence and sample-level privacy guarantees. Experiments on vision (CIFAR-10/100, Tiny-ImageNet with ResNet/ViT/Swin) and language (GLUE with RoBERTa) benchmarks demonstrate DP-FedAdamW consistently surpasses SOTA DPFL baselines under the same privacy budgets, especially in highly non-IID regimes. DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models Supplementary Material 9 More Implementation Detail 9.1 More Results Swin-Tiny/Base on CIFAR-10. Table 10 reports the averaged test accuracy on CIFAR-10 for six different DPFL methods evaluated on Swin-Tiny and Swin-Base, under two data heterogeneity levels (Dirichlet \alpha{=}0.6 and \alpha{=}0.1 ). Overall, DP-FedAdamW consistently achieves the best performance, while Swin-Base is markedly more accurate than Swin-Tiny and all methods degrade when the data becomes more heterogeneous (smaller \alpha ). For Swin-Base with \alpha{=}0.1 , DP-FedAdamW attains 90.68\% test accuracy, outperforming DP-FedSAM ( 89.82\% ) and DP-FedAvg ( 88.23\% ), even though DP-FedSAM needs roughly twice the privacy budget. Table 10: Averaged test accuracy (%) comparison on CIFAR-10. Assuming DP-FedAvg is (\varepsilon,\delta) -DP, all other methods share this budget except DP-FedSAM, which is approximately (2\varepsilon,\delta) -DP. Method Swin-Tiny Swin-Base Privacy Budget \alpha{=}0.6 \alpha{=}0.1 \alpha{=}0.6 \alpha{=}0.1 DP-FedAvg 82.93_{\pm 0.53} 78.24_{\pm 0.59} 89.56_{\pm 0.47} 88.23_{\pm 0.44} (\varepsilon,\delta) DP-SCAFFOLD 82.85_{\pm 0.50} 78.53_{\pm 0.54} 90.07_{\pm 0.48} 89.08_{\pm 0.62} (\varepsilon,\delta) DP-FedAvg-LS 83.94_{\pm 0.62} 79.67_{\pm 0.68} 90.97_{\pm 0.52} 90.14_{\pm 0.56} (\varepsilon,\delta) DP-FedSAM 84.05_{\pm 0.51} 80.36_{\pm 0.57} 91.25_{\pm 0.54} 89.82_{\pm 0.60} (2\varepsilon,\delta) DP-LocalAdamW 83.73_{\pm 0.58} 79.28_{\pm 0.64} 90.84_{\pm 0.39} 89.08_{\pm 0.41} (\varepsilon,\delta) DP-FedAdamW \color[rgb]{1,0,0}{84.23}_{\pm 0.32} \color[rgb]{1,0,0}{81.23}_{\pm 0.39} \color[rgb]{1,0,0}{91.79}_{\pm 0.34} \color[rgb]{1,0,0}{90.68}_{\pm 0.38} (\varepsilon,\delta) Noise multiplier results. To further analyze the impact of privacy noise on algorithm performance, we evaluate different noise multipliers \sigma\in\{0.5,1.0,1.5,2.0\} on CIFAR-100 using the Swin-Base model under Dirichlet \alpha{=}0.6 and \alpha{=}0.1 . As shown in Table 11 , increasing the noise multiplier consistently degrades test accuracy for all methods, while DP-FedAdamW maintains the best performance across all \sigma . Although \sigma{=}0.5 yields the highest accuracy, it also consumes a much larger privacy budget. In contrast, setting \sigma{=}1 significantly strengthens the formal privacy guarantee while only causing a small drop in accuracy, whereas larger noise levels ( \sigma{\geq}1.5 ) lead to a much more pronounced loss in utility. Therefore, we adopt \sigma{=}1 as the default noise multiplier in our main experiments, as it offers a favorable privacy–utility trade-off. Table 11: Test accuracy (%) on CIFAR-100 with Swin-Base under different noise multipliers \sigma . Method \alpha=0.6 \alpha=0.1 \sigma{=}0.5 \sigma{=}1 \sigma{=}1.5 \sigma{=}2 \sigma{=}0.5 \sigma{=}1 \sigma{=}1.5 \sigma{=}2 DP-FedAvg 67.23_{\pm 0.33} 65.61_{\pm 0.27} 63.78_{\pm 0.34} 60.22_{\pm 0.64} 62.49_{\pm 0.26} 60.77_{\pm 0.32} 57.85_{\pm 0.48} 54.34_{\pm 0.79} DP-SCAFFOLD 70.41_{\pm 0.25} 68.03_{\pm 0.55} 64.64_{\pm 0.48} 63.14_{\pm 0.58} 66.72_{\pm 0.38} 65.34_{\pm 0.49} 61.73_{\pm 0.44} 58.23_{\pm 0.66} DP-FedAvg-LS 70.93_{\pm 0.40} 68.80_{\pm 0.52} 66.52_{\pm 0.43} 64.77_{\pm 0.63} 68.36_{\pm 0.39} 66.08_{\pm 0.56} 62.87_{\pm 0.52} 59.98_{\pm 0.58} DP-FedSAM 72.55_{\pm 0.22} 70.44_{\pm 0.57} 67.45_{\pm 0.33} 65.75_{\pm 0.69} 67.94_{\pm 0.51} 66.28_{\pm 0.64} 62.21_{\pm 0.60} 60.84_{\pm 0.71} DP-LocalAdamW 70.74_{\pm 0.13} 68.33_{\pm 0.33} 66.46_{\pm 0.53} 64.10_{\pm 0.60} 67.35_{\pm 0.31} 65.28_{\pm 0.39} 60.55_{\pm 0.43} 58.78_{\pm 0.74} DP-FedAdamW \color[rgb]{1,0,0}{73.62}_{\pm 0.36} \color[rgb]{1,0,0}{71.58}_{\pm 0.30} \color[rgb]{1,0,0}{68.52}_{\pm 0.45} \color[rgb]{1,0,0}{66.79}_{\pm 0.51} \color[rgb]{1,0,0}{69.22}_{\pm 0.37} \color[rgb]{1,0,0}{67.45}_{\pm 0.26} \color[rgb]{1,0,0}{63.28}_{\pm 0.42} \color[rgb]{1,0,0}{61.05}_{\pm 0.61} 9.2 Datasets Vision datasets. We evaluate our methods on three widely used image classification benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet in Table 12 . CIFAR-10 and CIFAR-100 28 each contain 60{,}000 natural images of size 32\times 32 , split into 50{,}000 training and 10{,}000 test examples. CIFAR-10 comprises 10 coarse-grained object categories, whereas CIFAR-100 has finer labeling with 100 fine-grained categories organized into 20 superclasses. Tiny-ImageNet 30 is a downsampled subset of ImageNet with 200 classes, where each class contains 500 training images and 50 validation images of size 64\times 64 . Table 12: Summary of the vision datasets used in our experiments. Dataset #Classes Image Size Train / Test CIFAR-10 28 10 32\times 32 50,000 / 10,000 CIFAR-100 28 100 32\times 32 50,000 / 10,000 Tiny-ImageNet 30 200 64\times 64 100,000 / 10,000 Language datasets. For natural language understanding, we experiment on four tasks from the GLUE 67 benchmark: SST-2, QQP, QNLI, and MNLI (Table 13 . SST-2 is a binary sentiment classification task over movie reviews. QQP is a paraphrase identification task that determines whether two Quora questions express the same semantic intent. QNLI is a binary classification task that predicts whether a candidate sentence contains the answer to a given question. MNLI is a three-way natural language inference benchmark (entailment, contradiction, neutral) covering multiple genres of text. We follow the standard GLUE data splits and evaluation protocols. Table 13: Summary of GLUE datasets used in our experiments. Dataset Task Type #Classes Train/Test SST-2 Sentiment Classification (binary) 2 67,349 / 1,821 QQP Duplicate Question Detection (binary) 2 363,846 / 390,965 QNLI Question-Answer NLI (binary) 2 104,743 / 5,463 MNLI Natural Language Inference (entailment) 3 392,702 / 9,815 9.3 Models We study three representative architecture families, covering five specific models in total, as summarized in Table 14 . (1) ResNet-18. We adopt a ResNet-18 with Group Normalization (GN), where all Batch Normalization layers are replaced by GN to avoid dependence on batch statistics, which is more suitable for federated and differentially private training. To adapt CIFAR-10/100, we follow standard practices and modify the stem by replacing the original 7{\times}7 convolution with a 3{\times}3 kernel and removing the initial downsampling layers (i.e., the stride-2 convolution and max-pooling layer). (2) ViT-Base. We use the standard Vision Transformer Base (ViT-Base) architecture, which splits an input image into non-overlapping 16{\times}16 patches and linearly embeds them into a sequence of tokens with a prepended class token. The token sequence is processed by 12 Transformer encoder layers with multi-head self-attention and MLP blocks, and the final class token representation is fed to a linear classifier to produce the output logits. (3) Swin-Tiny. Swin-Tiny is a hierarchical Vision Transformer that uses shifted window self-attention. The model consists of four stages with depths [2,2,6,2] , embedding dimensions [96,192,384,768] , and attention heads [3,6,12,24] . Images are partitioned into 4{\times}4 patches, and each stage applies window-based attention with window size 7 and an MLP ratio of 4 , using LayerNorm and relative positional bias. DropPath regularization with depth-scaled rates is used, and the final features are aggregated by global pooling and passed to a linear classifier. (4) Swin-Base. Swin-Base is a larger variant of Swin-Tiny, sharing the same hierarchical shifted-window design. It is configured with four stages of depths [2,2,18,2] , embedding dimensions [128,256,512,1024] , and attention heads [4,8,16,32] , using a patch size of 4{\times}4 , window size 7 , and MLP ratio 4 . Similar to Swin-Tiny, the final stage features are pooled globally and fed into a linear classifier to generate predictions. (5) RoBERTa-Base. We use the RoBERTa-Base architecture, which follows the standard Transformer encoder design with 12 layers, 12 attention heads, and a hidden size of 768. It employs byte-pair encoding (BPE) tokenization and is pretrained with a masked language modeling objective on large-scale corpora using dynamic masking. The final representation corresponding to the classification token is fed into a task-specific linear classifier. Table 14: Overview of the model architectures used in our experiments. “Depth” denotes the total number of layers (blocks for ResNet/Swin, encoder layers for ViT/RoBERTa), and “Stages” denotes the number of blocks per stage for hierarchical models. Model Family Depth Stages Width / Dim Heads ResNet-18 (GN) CNN 18 [2,2,2,2] [64,128,256,512] – ViT-Base ViT Transformer 12 – 768 hidden, 3072 MLP 12 / layer Swin-Tiny Swin Transformer 12 [2,2,6,2] [96,192,384,768] [3,6,12,24] Swin-Base Swin Transformer 24 [2,2,18,2] [128,256,512,1024] [4,8,16,32] RoBERTa-Base NLP Transformer 12 – 768 hidden, 3072 FFN 12 / layer Algorithm 2 DP-LocalAdamW 1: Initial initial model \bm{\theta}^{0} ; rounds T ; local steps K ; step size \eta ; weight decay \lambda ; AdamW (\beta_{1},\beta_{2},\epsilon) ; clip norm C ; noise multiplier \sigma ; S clients per round 2: for t=1 to T do 3: for selected clients i=1,\dots,S in parallel do 4: \bm{\theta}_{i}^{t,0}\leftarrow\bm{\theta}^{t} ; \bm{m}_{i}^{t,0}\leftarrow\bm{0} ; \bm{v}_{i}^{t,0}\leftarrow\bm{0} 5: for k=1 to K do 6: sample \mathcal{D}_{i}^{k}\subset\mathcal{D}_{i} of size \lfloor sR\rfloor 7: for each sample j\in\mathcal{D}_{i}^{k} do 8: g_{ij}\leftarrow\nabla f_{i}(\bm{\theta}_{i}^{t,k};\xi_{ij}) 9: \bar{g}_{ij}\leftarrow g_{ij}/\max(1,\|g_{ij}\|_{2}/C) 10: end for 11: \tilde{g}_{i}^{t,k}\leftarrow\frac{1}{sR}\sum_{j\in\mathcal{D}_{i}^{k}}\bar{g}_{ij}+\frac{C}{sR}\mathcal{N}\!\left(0,\sigma^{2}C^{2}I\right) 12: \bm{m}_{i}^{t,k}\leftarrow\beta_{1}\bm{m}_{i}^{t,k-1}+(1-\beta_{1})\,\tilde{g}_{i}^{t,k} 13: \bm{v}_{i}^{t,k}\leftarrow\beta_{2}\bm{v}_{i}^{t,k-1}+(1-\beta_{2})\,\tilde{g}_{i}^{t,k}\odot\tilde{g}_{i}^{t,k} 14: \hat{\bm{m}}_{i}^{t,k}\leftarrow\bm{m}_{i}^{t,k}/(1-\beta_{1}^{k}) 15: \hat{\bm{v}}_{i}^{t,k}\leftarrow\bm{v}_{i}^{t,k}/(1-\beta_{2}^{k}) 16: \bm{\theta}_{i}^{t,k+1}\leftarrow\bm{\theta}_{i}^{t,k}-\eta\,(\hat{\bm{m}}_{i}^{t,k}/(\sqrt{\hat{\bm{v}}_{i}^{t,k}}+\epsilon)-\lambda\bm{\theta}_{i}^{t,k}) 17: end for 18: Clients send (\bm{\theta}_{i}^{t,K}-\bm{\theta}_{i}^{t,0}) to server 19: end for 20: \bm{\theta}^{t+1}\leftarrow\bm{\theta}^{t}+\frac{1}{S}\sum_{i=1}^{S}(\bm{\theta}_{i}^{t,K}-\bm{\theta}_{i}^{t,0}) 21: Broadcast \bm{\theta}^{t+1} to clients 22: end for 9.4 Complete Configuration Federated setup. In all experiments, we fine-tune the models using parameter-efficient Low-Rank Adaptation ( LoRA ). We then detail the federated learning configurations adopted in our experiments. Table 15 specifies the settings for all vision benchmarks, including the number of clients N , communication rounds T , local update steps K , client participation rate l , local batch size sR , DP noise multiplier \sigma , failure probability \delta , and LoRA hyperparameters (r,\alpha_{L}) . Table 17 reports the corresponding configuration for the GLUE benchmarks with RoBERTa-Base and LoRA, using the same federated parameters while additionally documenting the maximum sequence length and dropout rate. Table 15: Federated configurations for vision models across CIFAR-10/100 and Tiny-ImageNet. We report the number of clients N , communication rounds T , local steps K , client participation rate l , local batch size sR , DP noise multiplier \sigma , failure probability \delta , and LoRA hyperparameters (r,\alpha_{L}) and dropout rate. Dataset Model N T K l sR \sigma \delta r (LoRA) \alpha_{L} (LoRA) Dropout CIFAR-10 ResNet-18 (GN) 50 300 50 0.2 50 1.0 10^{-5} 16 32 0.1 ViT-Base 50 100 20 0.1 16 1.0 10^{-5} 16 32 0.1 Swin-Tiny/Base 50 100 20 0.1 16 1.0 10^{-5} 16 32 0.1 CIFAR-100 ResNet-18 (GN) 50 300 50 0.2 50 1.0 10^{-5} 16 32 0.1 ViT-Base 50 100 20 0.1 16 1.0 10^{-5} 16 32 0.1 Swin-Tiny/Base 50 100 20 0.1 16 1.0 10^{-5} 16 32 0.1 Tiny-ImageNet Swin-Tiny/Base 50 100 20 0.1 16 1.0 10^{-6} 16 32 0.1 Table 16: Federated configurations for RoBERTa-Base with LoRA on GLUE tasks. We report the number of clients N , communication rounds T , local steps K , client participation rate l , local batch size sR , DP noise multiplier \sigma , failure probability \delta , and LoRA hyperparameters (r,\alpha_{L}) , as well as the maximum sequence length and dropout rate. Dataset Model N T K l sR \sigma \delta r (LoRA) \alpha_{L} (LoRA) Length_max Dropout SST-2 RoBERTa-Base 4 100 20 1.0 16 1.0 10^{-5} 16 32 128 0.1 QQP 4 100 20 1.0 16 1.0 10^{-6} 16 32 128 0.1 QNLI 4 100 20 1.0 16 1.0 10^{-5} 16 32 128 0.1 MNLI 4 100 20 1.0 16 1.0 10^{-6} 16 32 128 0.1 Other setup. DP-FedAvg, DP-SCAFFOLD, DP-FedAvg-LS, and DP-FedSAM adopt learning rates selected from { {10^{-2},3{\times}10^{-2},5{\times}10^{-2},10^{-1},3{\times}10^{-1}} } with a fixed weight decay of 0.001 . For DP-FedSAM, the SAM perturbation \rho is selected via grid search over \{0.01,\,0.05,\,0.1,\,0.5\} . For DP-LocalAdamW and DP-FedAdamW, the learning rate is chosen from { {10^{-4},2{\times}10^{-4},3{\times}10^{-4},5{\times}10^{-4},8{\times}10^{-4},10^{-3}} }, combined with weight decay in \{0.01,0.001\} and standard AdamW parameters (\beta_{1}{=}0.9,\beta_{2}{=}0.999) . We employ cosine learning-rate decay throughout, and DP-FedAdamW additionally uses parameter \gamma{=}0.5 and weight decay \lambda{=}0.01 . For all visual fine-tuning tasks, we initialize from official ImageNet-22K pre-trained weights to ensure consistency across methods. Across all tasks, the gradient clipping threshold is chosen by grid search over \{0.05,\,0.1,\,0.2,\,0.3,\,0.5\} , and we find that larger clipping values can trigger gradient explosion. To obtain a simple and robust configuration across all benchmarks, we therefore adopt a unified clipping threshold of C{=}0.1 . Table 17: Hyperparameter configuration of ResNet-18 (CIFAR-10/100) across different algorithms. Method Local Optimizer \eta \beta_{1} \beta_{2} Weight Decay DP-FedAvg SGD 10^{-1} - - 0.001 DP-SCAFFOLD SGD 10^{-1} - - 0.001 DP-FedAvg-LS SGD 10^{-1} - - 0.001 DP-FedSAM SAM 10^{-1} - - 0.001 DP-LocalAdamW AdamW 3{\times}10^{-4} 0.9 0.999 0.01 DP-FedAdamW AdamW 3{\times}10^{-4} 0.9 0.999 0.01 10 DP-LocalAdamW Algorithm For completeness, we provide in Algorithm 2 the full local training procedure of DP-LocalAdamW . 11 Theoretical Analysis Details 11.1 Proof of Theorem 1 and Convergence Analysis Assumption A.1 (Smoothness) . (Smoothness) The non-convex f_{i} is a L -smooth function for all i\in[m] , i.e., \|\nabla f_{i}(\bm{\theta}_{1})-\nabla f_{i}(\bm{\theta}_{2})\|\leq L\|\bm{\theta}_{1}-\bm{\theta}_{2}\| , for all \bm{\theta}_{1},\bm{\theta}_{2}\in\mathbb{R}^{d} . Assumption A.2 (Bounded Stochastic Gradient) . \bm{g}_{i}^{t}=\nabla f_{i}(\bm{\theta}_{i}^{t},\xi_{i}^{t}) computed by using a sampled mini-batch data \xi_{i}^{t} in the local client i is an unbiased estimator of \nabla f_{i} with bounded variance, i.e., \mathbb{E}_{\xi_{i}^{t}}[\bm{g}_{i}^{t}]=\nabla f_{i}(\bm{\theta}_{i}^{t}) and \mathbb{E}_{\xi_{i}^{t}}\|g_{i}^{t}-\nabla f_{i}(\bm{\theta}_{i}^{t})\|^{2}\leq\sigma_{l}^{2} , for all \bm{\theta}_{i}^{t}\in\mathbb{R}^{d} . Assumption A.3 (Bounded Stochastic Gradient II) . Each element of stochastic gradient \bm{g}_{i}^{t} is bounded, i.e., \|\bm{g}_{i}^{t}\|_{\infty}=\|f_{i}(\bm{\theta}_{i}^{t},\xi_{i}^{t})\|_{\infty}\leq G_{g} , for all \bm{\theta}_{i}^{t}\in\mathbb{R}^{d} and any sampled mini-batch data \xi_{i}^{t} . Assumption A.4 (Bounded Heterogeneity) . The gradient dissimilarity between clients is bounded: \|\nabla f_{i}(\bm{\theta}_{1})-\nabla f_{i}(\bm{\theta}_{2})\|^{2}\leq\sigma_{g}^{2} , for all \bm{\theta}\in\mathbb{R}^{d} . In this section, we give the theoretical analysis of our proposed DP-FedAdamW algorithm. Firstly we state some standard assumptions for the non-convex function F . We use the common gradient bound condition in our proof. Then we can upper bound G_{\vartheta} as: \displaystyle G_{\vartheta}\ \displaystyle=\left\{\|\vartheta_{i}^{t,k}\|_{\infty}\right\}=\left\{\|\frac{1}{\sqrt{(1-\beta_{1}^{k})^{2}/(1-\beta_{2}^{t})\bm{v}_{i}^{t,k}}+\epsilon}\|_{\infty}\right\} \displaystyle=\left\{\|\frac{1}{\sqrt{(1-\beta_{1}^{k})^{2}/(1-\beta_{2}^{t})(\beta_{2}\bm{v}^{t,k-1}_{i}+\left(1-\beta_{2}\right)\bm{g}^{t,k}_{i}\cdot\bm{g}^{t,k}_{i}})+\epsilon}\|_{\infty}\right\}. \sqrt{(1-\beta_{1}^{k})^{2}/(1-\beta_{2}^{t})(\beta_{2}\bm{v}^{t,k-1}_{i}+\left(1-\beta_{2}\right)\bm{g}^{t,k}_{i}\cdot\bm{g}^{t,k}_{i}} is bounded as: \|\sqrt{(1-\beta_{1}^{k})^{2}/(1-\beta_{2}^{t})(\beta_{2}\bm{v}^{t,k-1}_{i}+\left(1-\beta_{2}\right)\bm{g}^{t,k}_{i}\cdot\bm{g}^{t,k}_{i}}\|_{\infty}\leq G_{g}. (15) Thus we bound G_{\vartheta} as \frac{1}{G_{g}}\leq G_{\vartheta}\leq\frac{1}{\epsilon} . 11.2 Main Lemmas Lemma 1 . Suppose \left\{X_{1},\cdots,X_{\tau}\right\}\subset\mathbb{R}^{d} be random variables that are potentially dependent. If their marginal means and variances satisfy \mathbb{E}\left[X_{i}\right]=\mu_{i} and \mathbb{E}\left[\|X_{i}-\right. \left.\mu_{i}\|^{2}\right]\leq\sigma^{2} , then it holds that \mathbb{E}\left[\left\|\sum_{i=1}^{\tau}X_{i}\right\|^{2}\right]\leq\left\|\sum_{i=1}^{\tau}\mu_{i}\right\|^{2}+\tau^{2}\sigma^{2}. If they are correlated in the Markov way such that \mathbb{E}\left[X_{i}\mid X_{i-1},\cdots X_{1}\right]=\mu_{i} and \mathbb{E}\left[\left\|X_{i}-\mu_{i}\right\|^{2}\mid\right. \left.\mu_{i}\right]\leq\sigma^{2} , i.e., the variables \left\{X_{i}-\mu_{i}\right\} form a martingale. Then the following tighter bound holds: \mathbb{E}\left[\left\|\sum_{i=1}^{\tau}X_{i}\right\|^{2}\right]\leq 2\mathbb{E}\left[\left\|\sum_{i=1}^{\tau}\mu_{i}\right\|^{2}\right]+2\tau\sigma^{2}. Lemma 2 . Given vectors v_{1},\cdots,v_{N}\in\mathbb{R}^{d} and \bar{v}=\frac{1}{N}\sum_{i=1}^{N}v_{i} , if we sample \mathcal{S}\subset\{1,\cdots,N\} uniformly randomly such that |\mathcal{S}|=S , then it holds that \mathbb{E}\left[\left\|\frac{1}{S}\sum_{i\in\mathcal{S}}v_{i}\right\|^{2}\right]=\|\bar{v}\|^{2}+\frac{N-S}{S(N-1)}\frac{1}{N}\sum_{i=1}^{N}\left\|v_{i}-\bar{v}\right\|^{2}. Proof. Letting \mathbb{I}\{i\in\mathcal{S}\} be the indicator for the event i\in\mathcal{S}_{r} , we prove this lemma by direct calculation as follows: \displaystyle\mathbb{E}\left[\left\|\frac{1}{S}\sum_{i\in\mathcal{S}}v_{i}\right\|^{2}\right] \displaystyle=\mathbb{E}\left[\left\|\frac{1}{S}\sum_{i=1}^{N}v_{i}\mathbb{I}\{i\in\mathcal{S}\}\right\|^{2}\right] \displaystyle=\frac{1}{S^{2}}\mathbb{E}\left[\left(\sum_{i}\left\|v_{i}\right\|^{2}\mathbb{I}\{i\in\mathcal{S}\}+2\sum_{i1 , \displaystyle\mathcal{E}_{t}= \displaystyle\mathbb{E}\left[\left\|\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-{\Delta}^{t+1}_{G}\right\|^{2}\right] \displaystyle= \displaystyle\mathbb{E}\left[\left\|(1-\gamma)\left(\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-{\Delta}^{t}_{G}\right)+\gamma\left(\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}g_{i}^{t,k}\odot\vartheta_{i}^{t,k}\right)\right\|^{2}\right] \displaystyle\leq \displaystyle\mathbb{E}\left[\left\|(1-\gamma)\left(\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-{\Delta}^{t}_{G}\right)\right\|^{2}\right]+\gamma^{2}\frac{1}{\epsilon^{2}}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)-\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}g_{i}^{t,k}\right\|^{2}\right] \displaystyle+2\gamma\mathbb{E}\left[\left\langle(1-\gamma)\left(\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-{\Delta}^{t}_{G}\right),\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}\nabla f\left(\bm{\theta}^{t}\right)\odot\vartheta_{i}^{t,k}-\frac{1}{SK}\sum_{i\in S^{t}}\sum_{k=1}^{K}g_{i}^{t,k}\odot\vartheta_{i}^{t,k}\right\rangle\right]. Note that \left\{\nabla F\left(\bm{\theta}_{i}^{t,k};\xi_{i}^{t,k}\right)\right\}_{0\leq k0 that \displaystyle\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{t}\right)\right\|^{2}\right] \displaystyle\leq(1+q)\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]+\left(1+q^{-1}\right)L^{2}\mathbb{E}\left[\left\|\bm{\theta}^{t}-\bm{\theta}^{t-1}\right\|^{2}\right] \displaystyle\leq(1+q)\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]+2\left(1+q^{-1}\right)\gamma^{2}L^{2}\left(\mathcal{E}_{t-1}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]\right) \displaystyle\leq(1+q)^{t}\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{0}\right)\right\|^{2}\right]+\frac{2}{q}\gamma^{2}L^{2}\sum_{j=0}^{t-1}\left(\mathcal{E}_{j}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{j}\right)\right\|^{2}\right)(1+q)^{t-j}\right.. Take q=\frac{1}{t} and we have \mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{t}\right)\right\|^{2}\right]\leq e\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{0}\right)\right\|^{2}\right]+2e(t+1)\gamma^{2}L^{2}\sum_{j=0}^{t-1}\left(\mathcal{E}_{j}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{j}\right)\right\|^{2}\right)\right.. (16) Note that this inequality is valid for t=0 . Therefore, using ( 16 ), we have \displaystyle\sum_{t=0}^{R-1}\Xi_{t}\leq \displaystyle\sum_{t=0}^{R-1}2\eta^{2}\mathbb{E}\left[(1-\gamma)^{2}\left\|{\Delta}^{t}_{G}\right\|^{2}+\gamma^{2}\frac{1}{\epsilon^{2}}\frac{1}{N}\sum_{i=1}^{N}\left\|\nabla f_{i}\left(\bm{\theta}^{t}\right)\right\|^{2}\right] \displaystyle\leq \displaystyle\sum_{t=0}^{R-1}2\eta^{2}\left(2(1-\gamma)^{2}\left(\mathcal{E}_{t-1}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]\right)+\gamma^{2}\frac{1}{\epsilon^{2}}\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{t}\right)\right\|^{2}\right]\right) \displaystyle\leq \displaystyle\sum_{t=0}^{R-1}4\eta^{2}(1-\gamma)^{2}\left(\mathcal{E}_{t-1}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]\right) \displaystyle+2\eta^{2}\gamma^{2}\frac{1}{\epsilon^{2}}\sum_{t=0}^{R-1}\left(\frac{e}{N}\sum_{i=1}^{N}\mathbb{E}\left[\left\|\nabla f_{i}\left(\bm{\theta}^{0}\right)\right\|^{2}\right]+2e(t+1)(\gamma L)^{2}\sum_{j=0}^{t-1}\left(\mathcal{E}_{j}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{j}\right)\right\|^{2}\right]\right)\right) \displaystyle\leq \displaystyle 4\eta^{2}(1-\gamma)^{2}\sum_{t=0}^{R-1}\left(\mathcal{E}_{t-1}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]\right) \displaystyle+2\eta^{2}\gamma^{2}\frac{1}{\epsilon^{2}}\left(eRG_{0}+2e(\gamma LR)^{2}\sum_{t=0}^{R-2}\left(\mathcal{E}_{t}+\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]\right)\right). Rearranging the equation and applying the upper bound of \eta , we finish the proof. ∎ Theorem 3 (Convergence for non-convex functions) . Under Assumption A.1 and A.2 and A.3 , if we take g^{0}=0 , \displaystyle\gamma=\min\left\{,\sqrt{\frac{SKL\Delta\epsilon^{2}}{\sigma_{l}^{2}R}}\right\}\text{ for any constant }c\in(0,1],\quad\gamma=\min\left\{\frac{1}{24LG_{g}},\frac{\gamma}{6L}\right\}, \displaystyle\eta KL\lesssim\min\left\{1,\frac{1}{\gamma\gamma LR},\left(\frac{L\Delta}{G_{0}\gamma^{3}R}\right)^{1/2},\frac{1}{(\gamma N)^{1/2}},\frac{1}{\left(\gamma^{3}NK\right)^{1/4}}\right\}. then DP-FedAdamW converges as \frac{1}{R}\sum_{t=0}^{R-1}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]\lesssim\sqrt{\frac{L\Delta\sigma_{l}^{2}}{SKR}}+\frac{L\Delta}{R}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}R^{2}}. Here G_{0}:=\frac{1}{N}\sum_{i=1}^{N}\left\|\nabla f_{i}\left(\bm{\theta}^{0}\right)\right\|^{2} . Proof. Combining Lemma 3 and 4 , we have \displaystyle\mathcal{E}_{t}\leq \displaystyle\left(1-\frac{8\gamma}{9}\right)\mathcal{E}_{t-1}+4\frac{(\gamma L)^{2}}{\gamma}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t-1}\right)\right\|^{2}\right]+\frac{2\gamma^{2}\sigma_{l}^{2}}{SK\epsilon^{2}} \displaystyle+4\frac{\gamma}{\epsilon^{2}}L^{2}\left(2eK^{2}\Xi_{t}+K\eta^{2}\gamma^{2}\frac{1}{\epsilon^{2}}\sigma_{l}^{2}\left(1+2K^{3}L^{2}\eta^{2}\gamma^{2}\right)\right., and \mathcal{E}_{0}\leq(1-\gamma)\mathcal{E}_{-1}+\frac{2\gamma^{2}\sigma_{l}^{2}}{SK\epsilon^{2}}+4\gamma\frac{1}{\epsilon^{2}}L^{2}\left(2eK^{2}\Xi_{0}+K\eta^{2}\gamma^{2}\frac{1}{\epsilon^{2}}\sigma_{l}^{2}\left(1+2K^{3}L^{2}\eta^{2}\gamma^{2}\right)\right). Summing over t from 0 to R-1 and applying Lemma 6 , \displaystyle\sum_{t=0}^{R-1}\mathcal{E}_{t}\leq \displaystyle\left(1-\frac{8\gamma}{9}\right)\sum_{t=-1}^{R-2}\mathcal{E}_{t}+4\frac{(\gamma L)^{2}}{\gamma}\sum_{t=0}^{R-2}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]+2\frac{\gamma^{2}\sigma_{l}^{2}}{SK\epsilon^{2}}R \displaystyle+4\gamma\frac{1}{\epsilon^{2}}L^{2}\left(2eK^{2}\sum_{t=0}^{R-1}\Xi_{t}+RK\eta^{2}\gamma^{2}\frac{1}{\epsilon^{2}}\sigma_{l}^{2}\left(1+2K^{3}L^{2}\eta^{2}\gamma^{2}\right)\right) \displaystyle\leq \displaystyle\left(1-\frac{7\gamma}{9}\right)\sum_{t=-1}^{R-2}\mathcal{E}_{t}+\left(4\frac{(\gamma L)^{2}}{\gamma}+\frac{\gamma}{9}\right)\sum_{t=-1}^{R-2}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]+16\gamma^{3}\frac{1}{\epsilon^{4}}(e\eta KL)^{2}RG_{0} \displaystyle+\frac{2\gamma^{2}\sigma_{l}^{2}}{SK\epsilon^{2}}R+4\gamma^{3}\frac{1}{\epsilon^{4}}(\eta KL)^{2}\left(\frac{1}{K}+2(\eta KL\gamma)^{2}\right)\sigma_{l}^{2}R \displaystyle\leq \displaystyle\left(1-\frac{7\gamma}{9}\right)\sum_{t=-1}^{R-2}\mathcal{E}_{t}+\frac{2\gamma}{9}\sum_{t=-1}^{R-2}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]+16\gamma^{3}\frac{1}{\epsilon^{4}}(e\eta KL)^{2}RG_{0}+\frac{4\gamma^{2}\sigma_{l}^{2}}{SK\epsilon^{2}}R. Here in the last inequality we apply 4\gamma\frac{1}{\epsilon^{4}}(\eta KL)^{2}\left(\frac{1}{K}+2(\eta KL\gamma)^{2}\right)\leq\frac{2}{NK}\quad\text{ and }\quad\gamma L\leq\frac{\gamma}{6}. Therefore, \sum_{t=0}^{R-1}\mathcal{E}_{t}\leq\frac{9}{7\gamma}\mathcal{E}_{-1}+\frac{2}{7}\mathbb{E}\left[\sum_{t=-1}^{R-2}\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]+\frac{144}{7}\frac{1}{\epsilon^{4}}(e\gamma\eta KL)^{2}G_{0}R+\frac{36\gamma\sigma_{l}^{2}}{7SK\epsilon^{2}}R. Combine this inequality with Lemma 3 and we get \frac{1}{\gamma}\mathbb{E}\left[f\left(\bm{\theta}^{t}\right)-f\left(\bm{\theta}^{0}\right)\right]\leq-\frac{1}{7}\sum_{t=0}^{R-1}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right]+\frac{39}{56\gamma}\mathcal{E}_{-1}+\frac{78}{7}\frac{1}{\epsilon^{4}}(e\gamma\eta KL)^{2}G_{0}R+\frac{39\gamma\sigma_{l}^{2}}{14SK\epsilon^{2}}R. Finally, noticing that g^{0}=0 implies \mathcal{E}_{-1}\leq 2L\left(f\left(\bm{\theta}^{0}\right)-f^{*}\right)=2L\Delta , we obtain \displaystyle\frac{1}{R}\sum_{t=0}^{R-1}\mathbb{E}\left[\left\|\nabla f\left(\bm{\theta}^{t}\right)\right\|^{2}\right] \displaystyle\lesssim\frac{L\Delta}{\gamma LR}+\frac{\mathcal{E}_{-1}}{\gamma T}+(\gamma\eta KL)^{2}\frac{1}{\epsilon^{4}}G_{0}+\frac{\gamma\sigma_{l}^{2}}{SK\epsilon^{2}}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}T^{2}} \displaystyle\lesssim\frac{L\Delta}{T}+\frac{L\Delta}{\gamma T}+\frac{\gamma\sigma_{l}^{2}}{SK\epsilon^{2}}+(\gamma\eta KL)^{2}G_{0}\frac{1}{\epsilon^{4}}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}T^{2}} \displaystyle\lesssim\frac{L\Delta}{T}+\sqrt{\frac{L\Delta\sigma_{l}^{2}}{SKT\epsilon^{2}}}+\frac{\sigma^{2}G_{g}^{2}}{s^{2}T^{2}}. ∎