Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data Junkang Liu, Fanhua Shang ∗, Hongying Liu ∗, Jin Liu, Weixin An, Yuanyuan Liu ∗ Abstract Second-order optimizers can significantly accelerate large-scale training, yet their naive federated variants are often unstable or even diverge on non-IID data. We show that a key culprit is preconditioner drift : client-side second-order training induces heterogeneous curvature-defined geometries (i.e., preconditioner coordinate systems), and server-side model averaging updates computed under incompatible metrics, corrupting the global descent direction. To address this geometric mismatch, we propose FedPAC , a preconditioner alignment and correction framework for reliable federated second-order optimization. FedPAC explicitly decouples parameter aggregation from geometry synchronization by: (i) Alignment (i.e.,aggregating local preconditioners into a global reference and warm-starting clients via global preconditioner); and (ii) Correction (i.e., steering local preconditioned updates using a global preconditioned direction to suppress long-term drift). We provide drift-coupled non-convex convergence guarantees with linear speedup under partial participation. Empirically, FedPAC consistently improves stability and accuracy across vision and language tasks, achieving up to 5.8 % 5.8\% absolute accuracy gain on CIFAR-100 with ViTs. Code is available at https://anonymous.4open.science/r/FedPAC-8B24 . 1 Introduction In recent years, the training paradigm of large-scale models, especially large language models (LLMs) ( 3 ) , has shifted significantly. It is increasingly recognized that the optimizer itself is part of the computational budget: under a fixed compute budget, better optimization methods can directly yield shorter training time and higher model quality. Second-order optimizers, such as Sophia ( 15 ) , SOAP ( 23 ) , and Muon ( 10 ) , leverage richer curvature information and have demonstrated substantial acceleration in centralized large-scale training, achieving 1.5–2× faster convergence over AdamW and SGD ( 1 ; 19 ) . This evidence suggests that second-order optimization is no longer merely a theoretical luxury, but is rapidly becoming a practical backbone of large-scale deep learning. Figure 1 : (a) In non-IID FL, first-order methods converge slowly, inducing little client drift. (b) Second-order methods converge faster locally and thus drift toward local optima, causing the aggregated global model to deviate from global optimum. (c) FedPAC corrects local second-order updates, yielding faster convergence and a global model closer to the global optimum. (d–f) FedPAC accelerates Sophia, Muon and SOAP to train ResNet-18 on CIFAR-100. The x-axis denotes the number of communication rounds, and the y-axis denotes test accuracy. In contrast, another highly practical and increasingly important training paradigm, Federated Learning (FL) ( 20 ) , has remained largely confined to the first-order regime in terms of optimization. Most widely adopted federated algorithms, including FedAvg ( 20 ) , and various variance-reduced variants, rely on first-order SGD updates ( 2 ) . Although these methods partially mitigate data heterogeneity and communication constraints, they continue to suffer from evident limitations in convergence speed, scalability to large models. This discrepancy naturally raises the following question: (a) ResNet-18, IID (b) ResNet-18, non-IID (c) ViT-Tiny, IID (d) ViT-Tiny, non-IID Figure 2 : The x-axis denotes communication rounds, and the y-axis is test accuracy. (a, c): In FL on IID data, second-order optimizers converge significantly faster than SGD and AdamW for training ResNet and Transformer. (b),(d): However, on non-IID data, second-order optimizer (e.g., Local Muon) converges much more slowly and can even underperform first-order methods such as Local SGD. If second-order optimizers have already demonstrated their effectiveness in centralized large-scale training, why can we not achieve similar acceleration benefits in FL? A natural baseline is to employ Sophia/SOAP/Muon as the local optimizer on each client and aggregate parameters as in FedAvg named Local Sophia/SOAP/Muon, as shown in Figure 2 . Yet, on non-IID data ( 16 ) , such a naive federated adaptation exhibits a critical failure mode: it fails to deliver the expected acceleration and often results in slower convergence or even divergence as shown in Figure 2 . Our systematic experiments and mechanistic analysis indicate that this behavior is not mainly due to the computational cost of second-order information or insufficient client-side compute. Instead, it is driven by a previously underexamined phenomenon, which we refer to as preconditioner drift . In centralized training, second-order preconditioners (e.g., the matrix preconditioners implicit curvature structures exploited by SOAP and Muon) are continuously updated under a single data distribution. In federated learning, however, each client is exposed to its own non-IID data distribution. As clients perform multiple local updates, their preconditioners adapt toward the local geometry associated with their respective data distributions, which leads to the following effects as in Figure 1 : ( i ) The preconditioners across different clients gradually drift apart in both scale and orientation; ( ii ) The server aggregates only the model parameters via simple averaging, implicitly assuming that all clients employ similar preconditioners; ( iii ) The resulting global update direction effectively combines inconsistent local curvature estimates to update a shared model, which severely distorts the global optimization trajectory. From an intuitive perspective, each client makes effective progress under its own curvature-adapted coordinate system; however, when these trajectories are naively aggregated, the resulting global optimization can become highly inefficient as in Figure 1 . Consistent with this intuition, we observe that in non-IID data settings, directly applying second-order optimizers not only fails to replicate the acceleration achieved in centralized training, but may even perform worse than simple first-order methods as in Figure 2 . This observation gives rise to an urgent and natural question: Do second-order optimizers still hold genuine potential in data heterogeneous federated learning? We answer this question affirmatively. We demonstrate that second-order optimizers can achieve substantial acceleration in FL when equipped with a dedicated framework to explicitly mitigate preconditioner drift as in Figure 1 . Accordingly, we propose FedPAC ( Fe derate d P reconditioner A lignment and C orrection), a principled correction and acceleration framework for second-order federated optimization. In summary, main contributions are as follows: \bullet We identify a previously underexplored failure mode of second-order federated learning: preconditioner drift —the mismatch of client-side curvature-induced geometries. We formalize it with an explicit drift metric and empirically show it strongly correlates with degraded convergence. \bullet We develop a unified framework, FedPAC , that targets this geometric mismatch by decoupling parameter updates from preconditioner synchronization: clients perform efficient local second-order updates, while preconditioners are periodically aligned and local steps are corrected using lightweight global curvature statistics. \bullet We provide drift-coupled convergence guarantees under standard smoothness and bounded-heterogeneity assumptions, where the optimization error contains an explicit drift term . We show FedPAC provably reduces this term and achieves faster convergence than first-order FL and naive second-order baselines. 2 Related Work 2.1 First-order FL under data heterogeneity In heterogeneous federated learning, most improvements of first-order methods target mitigating client drift and convergence instability caused by non-IID data. FedProx ( 14 ) stabilizes local updates via a proximal regularization term, while SCAFFOLD ( 11 ) theoretically analyzes client drift in FedAvg and corrects it using server–client control variates. FedCM ( 24 ) leverages client momentum to stabilize updates. Nevertheless, these methods remain fundamentally first-order, operating solely at the gradient level without explicitly leveraging second-order geometry such as the Hessian or Gauss–Newton information, leaving considerable room to explore the potential of second-order optimization in heterogeneous federated learning. Our alignment operates on the preconditioner operators that define the local metric, not on first-order states such as momentum or control variates. 2.2 Second-order optimization methods Traditional second-order methods such as Newton’s method are often prohibitively expensive. Recently, several practical second-order or quasi-second-order optimizers have been shown to scale to large models in centralized deep learning, including Shampoo ( 6 ) , SOAP ( 23 ) , Muon ( 10 ) , and Sophia ( 15 ) . Shampoo ( 6 ) performs structured gradient preconditioning via Kronecker-factored approximations of the Hessian. SOAP ( 23 ) stabilizes such preconditioning by running AdamW in the induced feature basis. Muon ( 10 ) orthogonalizes gradient momentum using Newton–Schulz iterations. Sophia ( 15 ) leverages lightweight diagonal Hessian estimates with element-wise clipping. 2.3 Second-order optimization in FL The utilization of second-order information in FL is still in its early stages but is developing rapidly. Fed-Sophia ( 5 ) adapts the Sophia optimizer to the federated setting and proposes a communication-efficient second-order federated algorithm. FedMuon ( 17 ) introduces the matrix-orthogonalization-based Muon optimizer into FL. Our framework generalizes the alignment and correction principle in FedMuon, and thus the latter can be viewed as a special case of the former. FedPM ( 9 ) systematically studies the empirical behavior of the second-order classical Newton method in FL, which is computational overhead and not suitable for large-scale models. 3 The Proposed Unified Algorithm 3.1 Problem Setup FL aims to optimize global model with the collaboration local clients, i.e., minimizing the following population risk: F(\boldsymbol{x})=\frac{1}{N}\sum_{i=1}^{N}\left(F_{i}(\boldsymbol{x}):=\mathbb{E}_{\xi_{i}\sim\mathcal{D}_{i}}\left[F_{i}\left(\boldsymbol{x};\xi_{i}\right)\right]\right). (1) The function F_{i} is the loss function on client i . \mathbb{E}_{\xi_{i}\sim\mathcal{D}_{i}}[\cdot] denotes conditional expectation with respect to the sample \xi_{i} . N is the number of clients, and \boldsymbol{x} is global model. (a) Local SOAP (b) FedPAC_SOAP (c) Local SOAP (d) FedPAC_SOAP Figure 3 : Performance and Preconditioner Drift of Local SOAP, FedPAC_SOAP . For SOAP, we compute the preconditioner drift in a layer-wise manner by measuring the spectral norm of the difference between each client’s left/right preconditioner and the aggregated global preconditioner. Our FedPAC_SOAP substantially reduces the preconditioner drift of Local SOAP and accelerates convergence in (a,b), reaching 40% test accuracy in fewer rounds, as shown in (c) and (d). Algorithm 1 Federated Second-Order Algorithm (FedSOA) 0: Per client, we maintain \boldsymbol{\Theta}_{i}^{r,k} within a round. Hyperparameters: learning rate \eta , communication rounds R , the number of local updates K , the number of client N . 1: for r=0,\dots,R do 2: for each client i\in\mathcal{S}_{r} in parallel do 3: \boldsymbol{\Theta}_{i}^{r,0}\leftarrow\boldsymbol{0} ; 4: for k=1,\dots,K do 5: Sample batch B_{i}^{r,k}; 6: \boldsymbol{g}_{i}^{r,k}\in\mathbb{R}^{m\times n}\leftarrow\nabla F_{i}(\boldsymbol{x}_{i}^{r,k};\xi_{i}^{r,k}); 7: \boldsymbol{\Theta}_{i}^{r,k+1}=\mathrm{UpdateState}\bigl(\boldsymbol{\Theta}_{i}^{r,k},\boldsymbol{g}_{i}^{r,k}\bigr) ; 8: \tilde{\boldsymbol{g}}_{i}^{r,k}\leftarrow\mathcal{P}_{\boldsymbol{\Theta}_{i}^{r,k}}\bigl(\boldsymbol{g}_{i}^{r,k}\bigr) ; 9: \boldsymbol{x}^{r,k+1}_{i}\!=\!\boldsymbol{x}^{r,k}_{i}\!\!-\!\eta_{l}\tilde{\boldsymbol{g}}_{i}^{r,k} ; 10: end for 11: \Delta\boldsymbol{x}_{i}^{r}\coloneqq\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}^{r} ; 12: Client i communicate (\boldsymbol{\Delta}\boldsymbol{x}_{i}^{r}) to Server; 13: end for 14: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}+\frac{1}{|\mathcal{S}_{r}|}\sum_{i=1}^{|\mathcal{S}_{r}|}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 15: end for 3.2 Unified Abstraction of Second-Order FL To investigate the mechanisms of second-order optimizers in FL, we first propose a generalized Federated Second-Order Algorithm (FedSOA), as shown in Algorithm 1 . At the k -th local step of round r , client i samples a mini-batch \xi_{i}^{r,k}\sim\mathcal{D}_{i} and computes a stochastic gradient \boldsymbol{g}_{i}^{r,k}=\nabla F_{i}(\boldsymbol{x}_{i}^{r,k};\xi_{i}^{r,k}). (2) Second-order optimizers maintain an internal preconditioner state \boldsymbol{\Theta}_{i}^{r,k} , which parameterizes a preconditioning operator that maps the gradient to an update direction. To unify different optimizers, we define a generic operator \mathcal{P}_{\boldsymbol{\Theta}}(\cdot) and write the local update as follows: \boldsymbol{x}_{i}^{r,k+1}=\boldsymbol{x}_{i}^{r,k}-\eta_{l}\,\mathcal{P}_{\boldsymbol{\Theta}_{i}^{r,k}}\bigl(\boldsymbol{g}_{i}^{r,k}\bigr), (3) where \eta_{l} denotes the local learning rate. The specific form of \mathcal{P}_{\boldsymbol{\Theta}} depends on the underlying optimizer. For example, SOAP apply structured preconditioning in a Kronecker-factored basis, Sophia rescales gradients using diagonal Hessian estimates, and Muon updates gradients in an orthogonalized rule. After each step, the preconditioner state is updated according to the optimizer rule: \boldsymbol{\Theta}_{i}^{r,k+1}=\mathrm{UpdateState}\bigl(\boldsymbol{\Theta}_{i}^{r,k},\boldsymbol{g}_{i}^{r,k}\bigr). (4) After K local steps, client i computes the model difference \Delta\boldsymbol{x}_{i}^{r}\coloneqq\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}^{r}. A naive second-order federated baseline aggregates these updates in the same manner as FedAvg: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}+\sum_{i\in\mathcal{S}_{r}}\frac{1}{|\mathcal{S}_{r}|}\,\Delta\boldsymbol{x}_{i}^{r}. (5) However, under strong statistical heterogeneity and multiple local steps, this naive aggregation leads to preconditioner drift , which significantly degrades global convergence. Three instantiations as (\boldsymbol{\Theta},\mathcal{P}) pairs. \bullet SOAP: \boldsymbol{\Theta}\!=\!\{L,R\},\mathcal{P}_{\boldsymbol{\Theta}}(g)\!\!=\!\!A\big(\beta_{1}m+(1-\beta_{1})g\big)B,A=\mathrm{InvRoot}(L+\epsilon I),B=\mathrm{InvRoot}(R+\epsilon I) . The \mathrm{InvRoot}(\cdot) denotes the inverse matrix p -th root operator. L corresponds to second-order information on the left dimension. R corresponds to second-order information on the right dimension. m denotes the momentum. \bullet Sophia: \boldsymbol{\Theta}=\{h\},\mathcal{P}_{\boldsymbol{\Theta}}(g)=\mathrm{clip}\!\left(\frac{\beta_{1}m+(1-\beta_{1})g}{h+\epsilon},\,\pm\rho\right). h denotes the curvature (second-order information) estimate. \mathrm{clip}(\cdot) denotes element-wise clipping. \bullet Muon: \boldsymbol{\Theta}\!=\!\{m\},\mathcal{P}_{\boldsymbol{\Theta}}(g)=\mathrm{Ortho}\left(\beta_{1}m+(1-\beta_{1})g\right). \mathrm{Ortho}(\cdot) denotes the orthogonalization operator. 3.3 Local Preconditioner Drift To characterize the geometric deviation of second-order optimization in federated environments, we consider an idealized centralized second-order optimizer that constructs a global preconditioner \Theta^{r} on the joint data distribution \mathcal{D}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{D}_{i} and performs the update \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}-\eta\,\mathcal{P}_{\Theta^{r}}\bigl(\nabla F(\boldsymbol{x}^{r})\bigr). (6) In this idealized and IID setting, the model always evolves under a unified second-order geometry characterized by \boldsymbol{\Theta}^{r} . In the federated setting, each client i locally updates its preconditioner state \Theta_{i}^{r,k} based on its own data distribution \mathcal{D}_{i} , which in turn induces a local preconditioner \boldsymbol{\Theta}_{i}^{r,k} . As k and r increase, these local preconditioners progressively drift across clients. We formally define the preconditioner drift \boldsymbol{\Delta}_{\mathcal{D}} , as the difference gap of the preconditioner drift between the global and local preconditioners. Definition 1 (Preconditioner drift) . \boldsymbol{\Delta}_{\mathcal{D}} of the global preconditioner \boldsymbol{\Theta}^{r+1}=\frac{1}{N}\sum_{i=1}^{N}\boldsymbol{\Theta}_{i}^{r,K} and the local preconditioners \left\{\boldsymbol{\Theta}_{i}^{r,K}\right\}_{i=1}^{N} is defined as: \boldsymbol{\Delta}_{\mathcal{D}}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}\left\|\boldsymbol{\Theta}_{i}^{r,K}-\boldsymbol{\Theta}^{r+1}\right\|^{2}. (7) Preconditioner drift is a client-level phenomenon: different clients’ preconditioners \boldsymbol{\Theta}_{i}^{r,k} gradually adapt toward their respective local geometries. As shown in Figure 3 (a,b), we observe that as data heterogeneity increases, the preconditioner drift \boldsymbol{\Delta}_{\mathcal{D}} becomes more severe, and the algorithm converges increasingly slowly. Algorithm 2 Federated Preconditioner Alignment and Correction Framework (FedPAC) 0: Per client, we maintain: \boldsymbol{\Theta}_{i}^{r,k} within a round. Hyperparameters: learning rate \eta , communication rounds R , the number of local updates K , the number of client N . 1: for r=0,\dots,R do 2: for each client i\in\mathcal{S}_{r} in parallel do 3: \boldsymbol{\Theta}_{i}^{r,0}\leftarrow\boldsymbol{\Theta}^{r}; 4: for k=1,\dots,K do 5: Sample batch B_{i}^{r,k} ; 6: \boldsymbol{g}_{i}^{r,k}\in\mathbb{R}^{m\times n}\leftarrow\nabla F_{i}(\boldsymbol{x}_{i}^{r,k};\xi_{i}^{r,k}) ; 7: \boldsymbol{\Theta}_{i}^{r,k+1}=\mathrm{UpdateState}\bigl(\boldsymbol{\Theta}_{i}^{r,k},\boldsymbol{g}_{i}^{r,k}\bigr); 8: \tilde{\boldsymbol{g}}_{i}^{r,k}\leftarrow\mathcal{P}_{\boldsymbol{\Theta}_{i}^{r,k}}\bigl(\boldsymbol{g}_{i}^{r,k}\bigr); 9: \boldsymbol{x}^{r,k+1}_{i}\!=\!\boldsymbol{x}^{r,k}_{i}\!\!-\!\eta_{l}[(1\!-\!\beta)\tilde{\boldsymbol{g}}_{i}^{r,k}\!+\!\beta\boldsymbol{g}_{G}^{r}] ; 10: end for 11: \Delta\boldsymbol{x}_{i}^{r}\coloneqq\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}^{r} ; 12: Client i communicate (\boldsymbol{\Delta}\boldsymbol{x}_{i}^{r},\boldsymbol{\Theta}_{i}^{r,K}) to Server; 13: end for 14: \boldsymbol{g}_{G}^{r+1}=-\frac{1}{SK\eta}\sum_{i=1}^{S}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 15: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}+\frac{1}{S}\sum_{i=1}^{S}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 16: \boldsymbol{\Theta}^{r+1}\;=\;\frac{1}{|\mathcal{S}_{r}|}\sum_{i\in\mathcal{S}_{r}}\boldsymbol{\Theta}_{i}^{r,K}; 17: Server broadcasts (\boldsymbol{x}^{r+1},\boldsymbol{\Theta}_{r+1},\boldsymbol{g}_{G}^{r+1}); 18: end for 4 Our Algorithm: A Federated Preconditioner Alignment and Correction Framework To address these issues, we propose a federated preconditioner alignment and correction framework ( FedPAC ) for general second-order optimizers, consisting of two stages. Figure 4 : An illustration of preconditioner drift, which corrects client drift through local-global alignment. (1) Aggregation of Preconditioner States (Alignment) At the end of round r , client i uploads \Delta\boldsymbol{x}_{i}^{r} and its local preconditioner state \boldsymbol{\Theta}_{i}^{r,K} ; the server then aggregates the preconditioners in Algorithm 2 (i.e., Lines 3 & 16): \boldsymbol{\Theta}^{r+1}\;=\;\frac{1}{|\mathcal{S}_{r}|}\sum_{i\in\mathcal{S}_{r}}\boldsymbol{\Theta}_{i}^{r,K}. (8) At the beginning of round r\!+\!1 , the server broadcasts (\boldsymbol{x}^{r},\boldsymbol{\Theta}^{r}) to the participating clients. Upon receiving them, client i aligns its local preconditioner state with the global reference \boldsymbol{\Theta}^{r} to obtain the aligned preconditioner used for the new round: \boldsymbol{\Theta}_{i}^{r,0}\leftarrow\boldsymbol{\Theta}^{r} . The client then uses \boldsymbol{\Theta}_{i}^{r,0} as the initial preconditioner state for round r\!+\!1 and performs the local second-order updates according to ( 3 ). This mechanism preserves local second-order adaptivity while periodically pulling the optimization geometry back toward a shared reference. (2) Local Preconditioner Correction (Correction) On each client, we correct the preconditioner drift, as follows: \boldsymbol{x}^{r,k+1}_{i}\!=\!\boldsymbol{x}^{r,k}_{i}\!\!-\!\eta_{l}[(1\!-\!\beta)\tilde{\boldsymbol{g}}_{i}^{r,k}\!+\!\beta\boldsymbol{g}_{G}^{r}], (9) where \boldsymbol{g}_{G}^{r}\!=\!-\frac{1}{SK\eta}\sum_{i=1}^{S}\big(\boldsymbol{x}_{i}^{r-1,K}\!-\!\boldsymbol{x}_{i}^{r-1,0}\big) is the estimated global update, obtained by averaging the preconditioned-gradient global change from the previous round. \beta is the trade-off coefficient between local and global updates in Figure 4 and Algorithm 2 (Lines 9 & 14). Overall, the above steps constitute a unified correction and acceleration framework for second-order federated optimization. The underlying local optimizer can be any structured second-order method, such as Sophia, Muon, SOAP. In this work, we instantiate our framework with three representative second-order optimizers, yielding FedPAC_Sophia , FedPAC_Muon , and FedPAC_SOAP , respectively. (a) CIFAR-100,Dir-0.1 (b) CIFAR-100,Dir-0.05 (c) Tiny-ImageNet,Dir-0.1 (d) Tiny-ImageNet,Dir-0.05 (e) CIFAR-100,Dir-0.1 (f) CIFAR-100,Dir-0.05 (g) Tiny-ImageNet,Dir-0.1 (h) Tiny-ImageNet,Dir-0.05 Figure 5 : Test accuracy versus communication rounds. The x-axis denotes communication rounds and the y-axis denotes the test accuracy. Panels (a–d) correspond to training with ResNet-18 , while panels (e–h) correspond to training with ViT-Tiny . Table 1 : Test accuracy of each method on CIFAR-100 and Tiny-Imagenet using ResNet-18 and ViT-Tiny over 300 communication rounds under Dir-0.1 and Dir-0.05 (100 clients, 10% participation, batch size 50, K=50 ). Method ResNet-18 ViT-Tiny CIFAR-100 Tiny-Imagenet CIFAR-100 Tiny-Imagenet Dir-0.1 Dir-0.05 Dir-0.1 Dir-0.05 Dir-0.1 Dir-0.05 Dir-0.1 Dir-0.05 FedAvg 60.17 56.75 47.48 43.80 27.24 23.42 15.68 14.05 SCAFFOLD 60.69 56.43 47.76 43.92 26.86 23.23 15.70 14.21 FedCM 66.61 62.65 41.16 36.00 16.95 14.74 8.88 8.15 Local AdamW 59.23 55.24 44.01 40.00 37.57 36.06 24.31 21.35 Local Sophia 56.65 50.89 41.23 36.15 34.05 32.25 22.49 21.14 FedPAC_Sophia 59.96 ( \uparrow 3.31 ) 53.66 ( \uparrow 2.77 ) 43.81 ( \uparrow 2.58 ) 36.37 ( \uparrow 0.22 ) 39.79 ( \uparrow 5.74 ) 32.71 ( \uparrow 0.46 ) 23.01 ( \uparrow 0.52 ) 22.37 ( \uparrow 1.23 ) Local Muon 67.26 49.86 52.83 34.76 44.00 39.68 30.51 28.25 FedPAC_Muon 71.85 ( \uparrow 4.59 ) 65.56 ( \uparrow 15.70 ) 57.95 ( \uparrow 5.12 ) 54.00 ( \uparrow 19.24 ) 47.81 ( \uparrow 3.81 ) 41.76 ( \uparrow 2.08 ) 31.45 ( \uparrow 0.94 ) 30.25 ( \uparrow 2.00 ) Local SOAP 68.44 58.16 54.42 50.02 49.41 41.68 33.30 30.36 FedPAC_SOAP 69.25 ( \uparrow 0.81 ) 64.16 ( \uparrow 6.00 ) 55.62 ( \uparrow 1.20 ) 51.81 ( \uparrow 1.79 ) 51.16 ( \uparrow 1.75 ) 47.55 ( \uparrow 5.87 ) 34.32 ( \uparrow 1.02 ) 31.33 ( \uparrow 0.97 ) 5 Theoretical Analysis In this section, we give the convergence theoretical analysis of our proposed FedPAC algorithm. Firstly, we state some standard assumptions for the non-convex function f . Assumption 5.1 (Smoothness) . The non-convex f_{i} is a L -smooth function for all i\in[N] , i.e., \|\nabla f_{i}(\boldsymbol{x})-\nabla f_{i}(\boldsymbol{y})\|\leq L\|\boldsymbol{x}-\boldsymbol{y}\| , for all \boldsymbol{x},\boldsymbol{y}\in\mathbb{R}^{d} . Assumption 5.2 (Bounded Stochastic Gradient) . \boldsymbol{g}_{i}^{r}=\nabla f_{i}(\boldsymbol{x}_{i}^{r},\xi_{i}^{r}) computed by using a sampled mini-batch data \xi_{i}^{r} in the local client i is an unbiased estimator of \nabla f_{i} with bounded variance, i.e., \mathbb{E}_{\xi_{i}^{r}}[\boldsymbol{g}_{i}^{r}]=\nabla f_{i}(\boldsymbol{x}_{i}^{r}) and \mathbb{E}_{\xi_{i}^{r}}\|\boldsymbol{g}_{i}^{r}-\nabla f_{i}(\boldsymbol{x}_{i}^{r})\|^{2}\leq\sigma_{l}^{2} , for all \boldsymbol{x}_{i}^{r}\in\mathbb{R}^{d} . Assumption 5.3 (Bounded Heterogeneity) . The dissimilarity between local clients is bounded on the gradients, i.e., \|\nabla f_{i}(\boldsymbol{x})-\nabla f(\boldsymbol{x})\|^{2}\leq\sigma_{g}^{2} , for all \boldsymbol{x}\in\mathbb{R}^{d} . Assumption 5.4 (Preconditioner Coercivity and Boundedness) . For any preconditioner state \Theta and any vector \boldsymbol{v}\in\mathbb{R}^{d} , the preconditioned mapping P_{\Theta}(\cdot) satisfies the following two properties: (i) ( coercivity ) \langle\boldsymbol{v},P_{\Theta}(\boldsymbol{v})\rangle\geq\mu\|\boldsymbol{v}\|^{2} for some constant \mu>0 ; (ii) ( boundedness ) \|P_{\Theta}(\boldsymbol{v})\|\leq M\|\boldsymbol{v}\| for some constant M>0 . Assumption 5.5 (Lipschitz Continuity in Preconditioner State) . There exists a constant L_{\boldsymbol{\Theta}}>0 such that for any two preconditioner states \boldsymbol{\Theta},\boldsymbol{\Theta}^{\prime} and any vector \boldsymbol{v}\in\mathbb{R}^{d} , \|P_{\boldsymbol{\Theta}}(\boldsymbol{v})-P_{\boldsymbol{\Theta}^{\prime}}(\boldsymbol{v})\|\leq L_{\boldsymbol{\Theta}}\|\boldsymbol{\Theta}-\boldsymbol{\Theta}^{\prime}\|\cdot\|\boldsymbol{v}\|. Our convergence theory holds for the class of state-dependent preconditioned mappings characterized by Assumptions 5.4 and 5.5 (spectrally bounded and Lipschitz in the preconditioner state), which is reasonably satisfied by SOAP/Muon/Sophia. Theorem 5.6 (Convergence of FedSOA for non-convex functions ) . Under Assumptions 5.1 , 5.2 , 5.3 , 5.4 , 5.5 , \exists\,G^{2}>0\ \text{s.t.}\ \sup_{r,i,k}\mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2}, if we take g^{0}=0 and choose the stepsize \eta such that \eta\;\leq\;\min\left\{\frac{\mu}{4LM^{2}K},\ \frac{1}{2LMK}\right\}, then FedSOA satisfies \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\big\|\nabla f(\boldsymbol{x}^{r})\big\|^{2}\!\lesssim\!\frac{L\Delta}{R}+\sqrt{\frac{L\Delta}{R}\!\cdot\!\frac{\sigma_{l}^{2}+\sigma_{g}^{2}+S\kappa_{\Theta}\,\bar{\Delta}_{D}}{SK}}, where \Delta=f(\boldsymbol{x}^{0})-f^{\star} , and \bar{\Delta}_{D}=\frac{1}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r},\Delta_{D}^{r}=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\big\|\Theta_{i}^{r,K}-\bar{\Theta}^{r,K}\big\|^{2}. and the drift-to-noise coupling constant is \kappa_{\Theta}\;:=\;L_{\Theta}^{2}\,G^{2}. Here S is the number of participating clients per round, K is the number of local iterations, and R is the total number of communication rounds. The notation \lesssim hides absolute constants depending only on (L,\mu,M) . Theorem 5.7 (Convergence of FedPAC for non-convex functions) . Under Assumptions 5.1 , 5.2 , 5.4 , and 5.5 , and the following second-moment bound \exists\,G^{2}>0\ \text{s.t.}\ \sup_{r,i,k}\mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2}, if we take g^{0}=0 and choose parameters such that \eta\;\leq\;\min\left\{\frac{\mu}{4LM^{2}K},\ \frac{1}{2LMK}\right\}, then FedPAC satisfies \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\big\|\nabla f(\boldsymbol{x}^{r})\big\|^{2}\;\lesssim\;\frac{L\Delta}{R}+\sqrt{\frac{L\Delta\,\sigma_{l}^{2}+\kappa_{\Theta}\cdot\bar{\Delta}_{D}}{SKR}}. (1) What limits second-order FL? Theorem 5.6 establishes the non-convex convergence of FedSOA and, more importantly, makes the preconditioner drift explicit: beyond the standard optimization term \frac{L\Delta}{R} and the stochastic/heterogeneity noise \frac{\sigma_{l}^{2}+\sigma_{g}^{2}}{SK} , an additional penalty S\kappa_{\Theta}\bar{\Delta}_{D} enters the effective noise. This shows that inconsistent client-side preconditioner states can dominate convergence even when gradients are well-estimated. (2) How does FedPAC close the gap? Theorem 5.7 shows that FedPAC eliminates the explicit heterogeneity term \sigma_{g}^{2} by aligning/correcting preconditioner dynamics, leaving only the drift-dependent remainder \bar{\Delta}_{D} . When \bar{\Delta}_{D} is small, FedPAC achieves near-ideal scaling dominated by \sigma_{l}^{2}/(SK) , making drift measurable and controllable in practice. 6 Experiments Datasets. We evaluate FedPAC on both vision and language tasks. ( i ) For image classification, we use CIFAR-100 ( 12 ) , and Tiny-ImageNet ( 13 ) . ( ii ) For NLP tasks, we adopt C4 ( 21 ) dataset. To simulate data heterogeneity across clients, we follow the Dirichlet partitioning scheme ( 8 ) . For Dir- \alpha partitioning, smaller \alpha indicates more severe data heterogeneity. Model Architectures. We explore a variety of model types: ( i ) ResNet-18 ( 7 ) as a representative convolutional neural network (CNN), ( ii ) Vision Transformer (ViT-Base) and ViT-Tiny ( 4 ) for Vision Transformers, and ( iii ) LLaMA ( 22 ) for large-scale language model. Baselines. We compare our method against state-of-the-art FL algorithms: FedAvg ( Local SGD ) ( 20 ) , SCAFFOLD ( 11 ) , FedCM ( 24 ) , Local AdamW , Local Sophia , Local Muon and Local SOAP . Under our FedPAC framework, we instantiate three variants: FedPAC_Sophia , FedPAC_Muon , and FedPAC_SOAP . In the Appendix (Table), we compare additional FL algorithms designed to address data heterogeneity. Hyperparameter Settings. For FedAvg , SCAFFOLD , FedCM , the lr is selected from \{10^{-2},\ 3\times 10^{-2},\ 5\times 10^{-2},\ 10^{-1},\ 3\times 10^{-1}\} , and choose the best value lr=10^{-1} . with a weight decay of 0.001 . For Local AdamW , the lr is selected from \{10^{-4},\ 3\times 10^{-4},\ 5\times 10^{-4},\ 8\times 10^{-4},\ 10^{-3}\} , and choose the best value lr=3\!\times\!10^{-4} . We set the learning rates for Local Sophia , Local Muon , and Local SOAP to 3\!\times\!10^{-4} , 3\!\times\!10^{-2} , and 3\!\times\!10^{-3} , respectively. Their weight decay is 0.01 . We apply cosine learning rate decay, and set FedPAC to \boldsymbol{\beta}\!=\!0.5 , weight decay 0.01 . We set the learning rate of FedPAC variants to be same with Local Sophia , Local Muon and Local SOAP . Additional hyperparameter configurations are detailed in the Appendix . We release all code, configuration files to ensure full reproducibility. All results are averaged over 5 runs with std reported with seeds 42, 43, 44, 45, 46. All experiments were performed on a single NVIDIA RTX 4090 GPU. Table 2 : Comparison of test accuracy and training loss for Vision Transformer (ViT-Base) under Dir-0.1 with 100 rounds (50 clients, 10% participation, batch size 16, K=50 ). Method CIFAR-100 Tiny-ImageNet Test Acc Loss Test Acc Loss FedAvg 89.28 0.332 86.97 0.421 SCAFFOLD 89.35 0.328 86.85 0.432 FedCM 87.44 0.592 84.24 0.708 Local AdamW 90.10 0.281 86.65 0.252 Local Sophia 90.21 0.275 86.16 0.188 FedPAC_Sophia 90.38 0.262 86.55 0.175 Local Muon 90.32 0.189 87.22 0.184 FedPAC_Muon 90.46 0.165 87.63 0.171 Local SOAP 90.44 0.151 87.72 0.212 FedPAC_SOAP 90.65 0.129 87.92 0.156 Table 3 : The train loss of each method on C4 data using LLaMA 60M , LLaMA 130M , LLaMA 350M over 100 communication rounds (20 clients, 20% participation, batch size 16, K=50 ). Method C4 (Train Loss) LLaMA 60M LLaMA 130M LLaMA 350M FedAvg 4.156 4.258 4.354 SCAFFOLD 4.028 4.238 4.365 FedCM 4.231 4.356 4.426 Local AdamW 3.213 3.418 3.519 Local Sophia 3.201 3.389 3.521 FedPAC_Sophia 3.188 3.367 3.485 Local Muon 3.156 3.256 3.365 FedPAC_Muon 3.145 3.229 3.352 Local SOAP 3.148 3.245 3.312 FedPAC_SOAP 3.121 3.215 3.341 Table 4 : Impact of \beta on FedPAC_SOAP with ViT-Tiny and ResNet-18 on CIFAR-100 under Dir-0.05. Model, Dataset, \beta 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 ResNet-18, CIFAR-100 62.05 62.35 63.10 63.57 63.75 64.16 63.75 61.75 61.70 58.86 ViT-Tiny, CIFAR-100 44.56 45.28 45.63 46.22 46.84 47.55 46.86 46.59 46.05 45.66 6.1 Results on Convolutional Neural Networks Training on CIFAR-100 with ResNet-18. Table 1 and Figure 1 present the test accuracy on CIFAR-100 and Tiny-ImageNet using ResNet-18. Accuracy under heterogeneity. Across both datasets, FedPAC consistently improves the corresponding local second-order optimizer, with the largest gains in highly non-IID regimes where preconditioner drift is severe. For example on CIFAR-100, Local Muon drops from 67.26\% (Dir-0.1) to 49.86\% (Dir-0.05), whereas FedPAC_Muon achieves 71.85\% and 65.56\% , respectively. FedPAC_SOAP also strengthens SOAP on CIFAR-100, improving from 68.44\% / 58.16\% ( Local SOAP ) to 69.25\% / 64.16\% under Dir-0.1/Dir-0.05. Similar trends hold on Tiny-ImageNet: Local Muon attains 52.83\% / 34.76\% (Dir-0.1/Dir-0.05), while FedPAC_Muon reaches 57.95\% / 54.00\% . Improvements for Sophia are also consistent (e.g., CIFAR-100: 56.65\%\rightarrow 59.96\% at Dir-0.1 and 50.89\%\rightarrow 53.66\% at Dir-0.05). Stability and drift mitigation. As illustrated in Figure 5 , FedPAC reduces preconditioner drift and yields faster, more stable convergence under strong heterogeneity, explaining why aggressive second-order methods (e.g., Muon/SOAP) remain effective in federated CNN training. 6.2 Results on Vision Transformer Training ViT-Tiny from scratch. Results are reported in Table 1 . FedPAC yields the strongest performance among second-order variants, especially under non-IID data. On CIFAR-100, FedPAC_SOAP improves over Local_SOAP from 49.41\%\!/41.68\% to 51.16\%\!/47.55\% under Dir- 0.1/0.05 , respectively. On Tiny-ImageNet, FedPAC_SOAP also achieves the best overall accuracy (e.g. 31.33\% at Dir- 0.05 ), while FedPAC_Muon shows clear robustness gains under heterogeneity (Dir- 0.1/0.05 : 31.45\%/30.25\% vs. 30.51\%/28.25\% for Local_Muon ). Overall, these results indicate that FedPAC effectively stabilizes federated ViT optimization and preserves accuracy as heterogeneity increases. Pretrained ViT-Base. Since small-scale datasets can limit ViT performance when trained from scratch, we additionally evaluate a pretrained ViT-Base under Dir- 0.1 (Table 2 ). FedPAC continues to provide consistent improvements: FedPAC_SOAP reaches 90.65\% on CIFAR-100 and 87.92\% on Tiny-ImageNet, surpassing the corresponding local second-order baselines (e.g., Local_SOAP : 90.44\%/87.72\% ). These results confirm the effectiveness of FedPAC on federated vision Transformers across both from-scratch and pretrained settings. 6.3 C4 federated pre-training with LLaMA. Table 3 summarizes C4 federated pre-training results after 100 rounds (20 clients, 20% participation) for LLaMA-60M/130M/350M. Standard FL optimizers struggle in this heterogeneous, partial-participation regime (e.g., FedAvg: 4.156/4.258/4.354), whereas locally using modern adaptive/second-order optimizers already yields large gains (e.g., Local SOAP: 3.148/3.245/3.312). FedPAC consistently matches or improves the strongest local baselines across scales, with the best results achieved by FedPAC_SOAP on 60M/130M (3.121/3.215) and competitive performance on 350M. Compared to FedAvg, the best configuration reduces training loss by \sim 1.0 absolute (e.g., 4.156 \rightarrow 3.121 on 60M), highlighting the practical benefit of stabilizing second-order methods in FL. 6.4 Ablation Study A1: Sensitivity to \beta (correction strength). Table 4 evaluates FedPAC_SOAP under Dir-0.05 on CIFAR-100. Performance is robust for moderate \beta and peaks around \beta{=}0.5 for both backbones (ResNet-18: 64.16 , ViT-Tiny: 47.55 ), indicating that a balanced global-direction correction is most effective. Too small \beta under-utilizes the correction signal (e.g., \beta{=}0 ), while overly large \beta over-regularizes toward the global direction and degrades accuracy (notably on ResNet-18 for \beta\geq 0.7 ). We thus use \beta{=}0.5 as the default in all experiments. A2: Component-wise ablation (Alignment vs. Correction). Table 5 shows that both modules are necessary. On Dir-0.05 CIFAR-100, Alignment-only and Correction-only each improves over Local SOAP , but Full FedPAC_SOAP achieves the best accuracy, indicating that alignment and correction are complementary. Table 5 : Ablation on FedPAC_SOAP components under Dir-0.05 on CIFAR-100. We report test accuracy after 300 rounds. Variant ResNet-18 ViT-Tiny Local SOAP 58.16 41.68 w/o preconditioner alignment ( \bar{\Theta} ) 61.12 43.67 w/o preconditioner correction ( \Delta_{G} ) 62.05 44.56 FedPAC_SOAP (full) \mathbf{64.16} \mathbf{47.55} A3: Communication-efficient preconditioner aggregation. Table 6 studies a compressed variant, FedPAC_SOAP_light , which uploads the SOAP preconditioner using SVD compression. While full FedPAC_SOAP improves accuracy over Local SOAP (47.55 vs. 41.68) at the cost of 3\times communication (68.4 MB/round), FedPAC_SOAP_light largely preserves the gain (46.75) with near-local bandwidth (25.1 MB/round, 1.1\times ). Notably, the computation overhead remains small across variants (5.56–5.91 s/round), indicating that SVD-based compression offers a favorable accuracy–communication trade-off. Table 6 : Communication-efficient preconditioner aggregation for SOAP on CIFAR-100, ViT-Tiny under Dir-0.05. CommCost denotes communication cost per round (MB), and CompCost denotes computation time per round (s). Aggregation Acc CommCost CompCost Local SOAP 41.68 22.8 MB (1 \times ) 5.56 s FedPAC_SOAP 47.55 68.4 MB (3 \times ) 5.85 s FedPAC_SOAP_light 46.75 25.1 MB (1.1 \times ) 5.91s 7 Conclusion We study how modern second-order optimizers behave in federated learning under non-IID data. We identify preconditioner drift as a key source of instability: after multiple local steps, clients adapt preconditioners to different local geometries, and naive aggregation across mismatched geometries distorts global updates. To mitigate this issue, we propose FedPAC , a preconditioner alignment and correction framework that aggregates and broadcasts a global reference preconditioner to align local geometry and suppress drift accumulation. We provide convergence guarantees for non-convex objectives and demonstrate consistent improvements in stability and accuracy on heterogeneous vision benchmarks across SOAP, Sophia, and Muon. 8 Limitations and Future Work Communication. Synchronizing preconditioner states increases communication, especially for large layers. Future work will explore efficient transmission/aggregation via low-rank compression. IID regimes. When data are close to IID, the alignment/correction may bring limited benefit and can underperform the original optimizer. Computation. Second-order local updates add computation overhead (about 1.2\times – 1.5\times vs. first-order), which may hinder deployment on constrained clients. More efficient approximations and system optimizations are needed. Impact Statement This work proposes FedPAC, a federated optimization framework that improves the stability and efficiency of curvature-aware (preconditioned/second-order) training under client heterogeneity and partial participation. More reliable and communication-efficient federated training can lower resource and energy costs and broaden access to collaborative learning in domains where data cannot be centralized (e.g., on-device personalization or cross-silo settings). However, stronger federated training may also amplify risks from inappropriate deployment, including unfair decisions, privacy leakage, or misuse. Federated learning does not by itself guarantee privacy or security, and our method provides no formal privacy guarantees; practitioners should combine it with established protections (e.g., secure aggregation, differential privacy, and auditing) and evaluate robustness and fairness prior to deployment. Appendix A Appendix A: Proof of Theorem 1 and Convergence Analysis A.1 FedPAC Algorithm A.2 Assumption We analyze generalization based on following assumptions: Assumption A.1 . (Smoothness). F_{i} is L -smooth for all i\in [N] , \left\|\nabla F_{i}(\boldsymbol{x}_{1})-\nabla F_{i}(\boldsymbol{x}_{2})\right\|\leq L\|\boldsymbol{x}_{1}-\boldsymbol{x}_{2}\| (10) for all \boldsymbol{x}_{1},\boldsymbol{x}_{2} in its domain and i\in[N] . Assumption A.2 . (Bounded variance of data heterogeneity). The global variability of the local gradient of the loss function is bounded by \sigma_{g}^{2} for all i\in[N] , \left\|\nabla F_{i}\left(\boldsymbol{x}\right)-\nabla F\left(\boldsymbol{x}\right)\right\|^{2}\leq\sigma_{g}^{2} (11) Assumption A.3 . (Bounded variance of stochastic gradient). The stochastic gradient \nabla F_{i}\left(\boldsymbol{x},\xi_{i}\right) , computed by the i -th client of model parameter \boldsymbol{x} using mini-batch \xi_{i} , is an unbiased estimator of \nabla F_{i}(\boldsymbol{x}) with variance bounded by \sigma_{l}^{2} , i.e., \mathbb{E}_{\xi_{i}}\left\|\nabla F_{i}\left(\boldsymbol{x},\xi_{i}\right)-\nabla F_{i}(\boldsymbol{x})\right\|^{2}\leq\sigma_{l}^{2} (12) for all i\in[N] , where the expectation is over all local datasets. Assumption A.4 (Preconditioner Coercivity and Boundedness) . For any preconditioner state \Theta and any vector \boldsymbol{v}\in\mathbb{R}^{d} , the preconditioned mapping P_{\Theta}(\cdot) satisfies the following two properties: (i) ( coercivity ) \langle\boldsymbol{v},P_{\Theta}(\boldsymbol{v})\rangle\geq\mu\|\boldsymbol{v}\|^{2} for some constant \mu>0 ; (ii) ( boundedness ) \|P_{\Theta}(\boldsymbol{v})\|\leq M\|\boldsymbol{v}\| for some constant M>0 . Assumption A.5 (Lipschitz Continuity in Preconditioner State) . There exists a constant L_{\Theta}>0 such that for any two preconditioner states \Theta,\Theta^{\prime} and any vector \boldsymbol{v}\in\mathbb{R}^{d} , \|P_{\Theta}(\boldsymbol{v})-P_{\Theta^{\prime}}(\boldsymbol{v})\|\leq L_{\Theta}\|\Theta-\Theta^{\prime}\|\cdot\|\boldsymbol{v}\|. Appendix B Proof of Drift-Coupled Convergence for FedSOA and Alignment to Theorem 5.6 B.1 Setup and Notation Let F(x):=\frac{1}{N}\sum_{i=1}^{N}F_{i}(x) be a (possibly non-convex) objective. In communication round r , a set S_{r} of S clients participates. Each client performs K local updates: x_{i}^{r,k+1}=x_{i}^{r,k}-\eta\tilde{g}_{i}^{r,k},\qquad\tilde{g}_{i}^{r,k}:=P_{\Theta_{i}^{r,k}}\!\big(g_{i}^{r,k}\big),\qquad g_{i}^{r,k}:=\nabla F_{i}(x_{i}^{r,k};\xi_{i}^{r,k}). (13) The server aggregates: x^{r+1}=x^{r}+\frac{1}{S}\sum_{i\in S_{r}}\big(x_{i}^{r,K}-x_{i}^{r,0}\big)=x^{r}-\eta\,U^{r},\qquad U^{r}:=\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\tilde{g}_{i}^{r,k}. (14) Define the (per-round) averaged preconditioner state \bar{\Theta}^{r,K}:=\frac{1}{S}\sum_{i\in S_{r}}\Theta_{i}^{r,K} and drift metric \Delta_{D}^{r}:=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\big\|\Theta_{i}^{r,K}-\bar{\Theta}^{r,K}\big\|^{2}. (15) B.2 Assumptions (A1) Smoothness. Each F_{i} is L -smooth: \|\nabla F_{i}(x)-\nabla F_{i}(y)\|\leq L\|x-y\| . (A2) Unbiased stochastic gradients with bounded variance. \mathbb{E}[g_{i}^{r,k}\mid x_{i}^{r,k}]=\nabla F_{i}(x_{i}^{r,k}) and \mathbb{E}\|g_{i}^{r,k}-\nabla F_{i}(x_{i}^{r,k})\|^{2}\leq\sigma_{l}^{2} . (A3) Bounded heterogeneity (FedSOA only). \|\nabla F_{i}(x)-\nabla F(x)\|^{2}\leq\sigma_{g}^{2} for all i,x . (P1) Preconditioner coercivity and boundedness. There exist constants 0<\mu\leq M such that for all \Theta and all v , \langle v,P_{\Theta}(v)\rangle\geq\mu\|v\|^{2},\qquad\|P_{\Theta}(v)\|\leq M\|v\|. (16) (P2) Lipschitz continuity in the preconditioner state. There exists L_{\Theta}>0 such that for all \Theta,\Theta^{\prime} and all v , \|P_{\Theta}(v)-P_{\Theta^{\prime}}(v)\|\leq L_{\Theta}\|\Theta-\Theta^{\prime}\|\cdot\|v\|. (17) (A4) Second-moment bound. There exists G^{2} such that \sup_{r,i,k}\mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2} . B.3 A Key Lemma: Injecting Drift into the Upper Bound Lemma B.1 (Drift-induced preconditioned disagreement) . Let \bar{\Theta} be any reference state and \{\Theta_{i}\}_{i\in S_{r}} be local states. Then for any vectors \{v_{i}\}_{i\in S_{r}} , \left\|\frac{1}{S}\sum_{i\in S_{r}}\big(P_{\Theta_{i}}(v_{i})-P_{\bar{\Theta}}(v_{i})\big)\right\|^{2}\leq L_{\Theta}^{2}\left(\frac{1}{S}\sum_{i\in S_{r}}\|v_{i}\|^{2}\right)\left(\frac{1}{S}\sum_{i\in S_{r}}\|\Theta_{i}-\bar{\Theta}\|^{2}\right). (18) Proof. By ( 17 ), \left\|\frac{1}{S}\sum_{i}(P_{\Theta_{i}}(v_{i})-P_{\bar{\Theta}}(v_{i}))\right\|\leq\frac{1}{S}\sum_{i}L_{\Theta}\|\Theta_{i}-\bar{\Theta}\|\cdot\|v_{i}\|. Apply Cauchy–Schwarz: \Big(\frac{1}{S}\sum_{i}a_{i}b_{i}\Big)^{2}\leq\Big(\frac{1}{S}\sum_{i}a_{i}^{2}\Big)\Big(\frac{1}{S}\sum_{i}b_{i}^{2}\Big), with a_{i}=\|\Theta_{i}-\bar{\Theta}\| and b_{i}=\|v_{i}\| . ∎ Instantiation with \Delta_{D}^{r} . Take \bar{\Theta}=\bar{\Theta}^{r,K} and \Theta_{i}=\Theta_{i}^{r,K} , then \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\big(P_{\Theta_{i}^{r,K}}(v_{i})-P_{\bar{\Theta}^{r,K}}(v_{i})\big)\right\|^{2}\leq L_{\Theta}^{2}\left(\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|v_{i}\|^{2}\right)\Delta_{D}^{r}. (19) B.4 A Decomposition of the Preconditioned Direction Fix the reference state \bar{\Theta}^{r,K} and decompose \tilde{g}_{i}^{r,k} as \displaystyle\tilde{g}_{i}^{r,k} \displaystyle=P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big) \displaystyle\quad+\underbrace{\Big(P_{\bar{\Theta}^{r,K}}\!\big(\nabla F_{i}(x^{r})\big)-P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)\Big)}_{\textsf{heterogeneity term}} \displaystyle\quad+\underbrace{\Big(P_{\bar{\Theta}^{r,K}}\!\big(g_{i}^{r,k}\big)-P_{\bar{\Theta}^{r,K}}\!\big(\nabla F_{i}(x_{i}^{r,k})\big)\Big)}_{\textsf{stochastic/local-drift term}} \displaystyle\quad+\underbrace{\Big(P_{\Theta_{i}^{r,k}}\!\big(g_{i}^{r,k}\big)-P_{\bar{\Theta}^{r,K}}\!\big(g_{i}^{r,k}\big)\Big)}_{\textsf{precond drift term}}. (20) Define the aggregated error E^{r}:=\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\Big[\textsf{hetero}_{i}^{r,k}+\textsf{stoch}_{i}^{r,k}+\textsf{pdrift}_{i}^{r,k}\Big],\qquad U^{r}=K\,P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)+E^{r}. (21) We first recall the definition of the server-aggregated direction: U^{r}\;:=\;\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\tilde{g}_{i}^{r,k}. (22) Fix a reference preconditioner state \bar{\Theta}^{r,K} . Using the decomposition in ( 21 ), i.e., \tilde{g}_{i}^{r,k}=P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)+\textsf{hetero}_{i}^{r,k}+\textsf{stoch}_{i}^{r,k}+\textsf{pdrift}_{i}^{r,k}, (23) and substituting ( 23 ) into ( 22 ), we obtain \displaystyle U^{r} \displaystyle=\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\Big[P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)+\textsf{hetero}_{i}^{r,k}+\textsf{stoch}_{i}^{r,k}+\textsf{pdrift}_{i}^{r,k}\Big] \displaystyle=\underbrace{\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)}_{:=\,(\star)}\;+\;\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\Big[\textsf{hetero}_{i}^{r,k}+\textsf{stoch}_{i}^{r,k}+\textsf{pdrift}_{i}^{r,k}\Big]. (24) Since P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big) does not depend on i nor k , we simplify (\star) as (\star)=\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)=\frac{1}{S}\sum_{i\in S_{r}}\Big(\sum_{k=0}^{K-1}1\Big)\,P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)=K\,P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big). (25) Define the aggregated error term E^{r}\;:=\;\frac{1}{S}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\Big[\textsf{hetero}_{i}^{r,k}+\textsf{stoch}_{i}^{r,k}+\textsf{pdrift}_{i}^{r,k}\Big]. (26) Substituting ( 25 ) and ( 26 ) into ( 24 ) yields U^{r}=K\,P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)+E^{r}. (27) B.5 One-Step Descent Inequality By L -smoothness of F , F(x^{r+1})\leq F(x^{r})+\langle\nabla F(x^{r}),x^{r+1}-x^{r}\rangle+\frac{L}{2}\|x^{r+1}-x^{r}\|^{2}. (28) Using ( 14 ), x^{r+1}-x^{r}=-\eta U^{r} , F(x^{r+1})\leq F(x^{r})-\eta\langle\nabla F(x^{r}),U^{r}\rangle+\frac{L\eta^{2}}{2}\|U^{r}\|^{2}. (29) Plug ( 21 ) into the inner product: \displaystyle-\eta\langle\nabla F(x^{r}),U^{r}\rangle \displaystyle=-\eta K\left\langle\nabla F(x^{r}),P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)\right\rangle-\eta\langle\nabla F(x^{r}),E^{r}\rangle \displaystyle\leq-\eta K\mu\|\nabla F(x^{r})\|^{2}-\eta\langle\nabla F(x^{r}),E^{r}\rangle, (30) where we used ( 16 ). For the cross term, apply Young’s inequality with parameter \mu/2 : -\eta\langle\nabla F(x^{r}),E^{r}\rangle\leq\eta\cdot\frac{\mu}{4}\|\nabla F(x^{r})\|^{2}+\eta\cdot\frac{1}{\mu}\|E^{r}\|^{2}. (31) Combining ( 29 )–( 31 ), F(x^{r+1})\leq F(x^{r})-\eta K\frac{3\mu}{4}\|\nabla F(x^{r})\|^{2}+\eta\frac{1}{\mu}\|E^{r}\|^{2}+\frac{L\eta^{2}}{2}\|U^{r}\|^{2}. (32) B.6 Bounding \mathbb{E}\|E^{r}\|^{2} and \mathbb{E}\|U^{r}\|^{2} We bound three components in E^{r} . (i) Heterogeneity term. By ( 16 ) boundedness and (A3), \left\|P_{\bar{\Theta}^{r,K}}\!\big(\nabla F_{i}(x^{r})\big)-P_{\bar{\Theta}^{r,K}}\!\big(\nabla F(x^{r})\big)\right\|^{2}\leq M^{2}\|\nabla F_{i}(x^{r})-\nabla F(x^{r})\|^{2}\leq M^{2}\sigma_{g}^{2}. (33) (ii) Stochastic/local-drift term. By ( 16 ) boundedness and (A2), \displaystyle\mathbb{E}\left\|P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(\nabla F_{i}(x_{i}^{r,k}))\right\|^{2} \displaystyle\leq M^{2}\,\mathbb{E}\|g_{i}^{r,k}-\nabla F_{i}(x_{i}^{r,k})\|^{2} \displaystyle\leq M^{2}\sigma_{l}^{2}. (34) (Any additional “local model drift” term caused by x_{i}^{r,k}\neq x^{r} can be absorbed into the constant by standard arguments; we keep the presentation clean by grouping it into the stochastic/local-drift bucket.) (iii) Preconditioner drift term. Applying Lemma B.1 with v_{i}=g_{i}^{r,k} and the instantiation ( 19 ), together with (A4), yields \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2}\leq L_{\Theta}^{2}\,G^{2}\,\Delta_{D}^{r}. (35) Preconditioner drift term (detailed proof). Recall Lemma B.1 : for any reference state \bar{\Theta} , any local states \{\Theta_{i}\}_{i\in S_{r}} , and any vectors \{v_{i}\}_{i\in S_{r}} , \left\|\frac{1}{S}\sum_{i\in S_{r}}\big(P_{\Theta_{i}}(v_{i})-P_{\bar{\Theta}}(v_{i})\big)\right\|^{2}\leq L_{\Theta}^{2}\left(\frac{1}{S}\sum_{i\in S_{r}}\|v_{i}\|^{2}\right)\left(\frac{1}{S}\sum_{i\in S_{r}}\|\Theta_{i}-\bar{\Theta}\|^{2}\right). (36) We instantiate ( 36 ) by choosing \Theta_{i}\leftarrow\Theta_{i}^{r,k},\qquad\bar{\Theta}\leftarrow\bar{\Theta}^{r,K},\qquad v_{i}\leftarrow g_{i}^{r,k}. Then ( 36 ) gives \displaystyle\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2} \displaystyle\leq L_{\Theta}^{2}\left(\frac{1}{S}\sum_{i\in S_{r}}\|g_{i}^{r,k}\|^{2}\right)\left(\frac{1}{S}\sum_{i\in S_{r}}\|\Theta_{i}^{r,k}-\bar{\Theta}^{r,K}\|^{2}\right). (37) Taking expectation on both sides and using linearity of expectation yields \displaystyle\mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2} \displaystyle\leq L_{\Theta}^{2}\,\mathbb{E}\left[\left(\frac{1}{S}\sum_{i\in S_{r}}\|g_{i}^{r,k}\|^{2}\right)\left(\frac{1}{S}\sum_{i\in S_{r}}\|\Theta_{i}^{r,k}-\bar{\Theta}^{r,K}\|^{2}\right)\right]. (38) Next, we bound the first factor by Assumption (A4): \mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2} for all i,r,k . Hence, \mathbb{E}\left(\frac{1}{S}\sum_{i\in S_{r}}\|g_{i}^{r,k}\|^{2}\right)=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2}. (39) Finally, to express the second factor in terms of the drift metric, we use the instantiation ( 19 ) with \bar{\Theta}=\bar{\Theta}^{r,K} . If the drift metric is defined at step k as \Delta_{D}^{r,k}:=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\big\|\Theta_{i}^{r,k}-\bar{\Theta}^{r,K}\big\|^{2}, (40) then ( 38 )–( 39 ) imply \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2}\leq L_{\Theta}^{2}\,G^{2}\,\Delta_{D}^{r,k}. (41) In particular, if we upper bound \Delta_{D}^{r,k}\leq\Delta_{D}^{r} for all k\in\{0,\dots,K-1\} (e.g., by monotonicity or by defining \Delta_{D}^{r}:=\max_{0\leq k\leq K}\Delta_{D}^{r,k} ), we obtain the stated bound: \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2}\leq L_{\Theta}^{2}\,G^{2}\,\Delta_{D}^{r}. (42) Remark (optional). If one prefers to keep the dependence on k , then ( 41 ) is the tight form, and the analysis can carry \Delta_{D}^{r,k} through the subsequent steps. Aggregate bound. Using \|\sum_{t=1}^{T}a_{t}\|^{2}\leq T\sum_{t=1}^{T}\|a_{t}\|^{2} and the fact that averaging over S clients reduces variance by S , we obtain (for some universal constant c>0 ) \mathbb{E}\|E^{r}\|^{2}\leq c\,K\left(\frac{M^{2}\sigma_{l}^{2}}{S}+\frac{M^{2}\sigma_{g}^{2}}{S}+L_{\Theta}^{2}G^{2}\Delta_{D}^{r}\right). (43) Moreover, by \|a+b\|^{2}\leq 2\|a\|^{2}+2\|b\|^{2} and ( 16 ), \displaystyle\mathbb{E}\|U^{r}\|^{2} \displaystyle=\mathbb{E}\left\|KP_{\bar{\Theta}^{r,K}}(\nabla F(x^{r}))+E^{r}\right\|^{2} \displaystyle\leq 2K^{2}\mathbb{E}\|P_{\bar{\Theta}^{r,K}}(\nabla F(x^{r}))\|^{2}+2\mathbb{E}\|E^{r}\|^{2} \displaystyle\leq 2K^{2}M^{2}\mathbb{E}\|\nabla F(x^{r})\|^{2}+2\mathbb{E}\|E^{r}\|^{2}. (44) B.7 Per-Round Recursion with an Explicit Drift Penalty Taking expectation of ( 32 ) and plugging ( 43 )–( 44 ), we obtain \displaystyle\mathbb{E}F(x^{r+1}) \displaystyle\leq\mathbb{E}F(x^{r})-\eta K\left(\frac{3\mu}{4}-L\eta KM^{2}\right)\mathbb{E}\|\nabla F(x^{r})\|^{2} \displaystyle\quad+c\,\eta\left(\frac{1}{\mu}+L\eta\right)K\left(\frac{M^{2}(\sigma_{l}^{2}+\sigma_{g}^{2})}{S}+L_{\Theta}^{2}G^{2}\Delta_{D}^{r}\right). (45) Choose \eta such that L\eta KM^{2}\leq\frac{\mu}{4} , i.e., \eta\leq\frac{\mu}{4LKM^{2}}. (46) Then the descent coefficient is positive and ( 45 ) simplifies to \mathbb{E}F(x^{r+1})\leq\mathbb{E}F(x^{r})-\frac{\mu}{2}\eta K\,\mathbb{E}\|\nabla F(x^{r})\|^{2}+C_{0}\,L\eta^{2}K\left(\frac{\sigma_{l}^{2}+\sigma_{g}^{2}}{S}+\underbrace{\frac{L_{\Theta}^{2}G^{2}}{M^{2}}}_{:=\;\kappa_{\Theta}}\Delta_{D}^{r}\right), (47) where C_{0}>0 is an absolute constant (absorbing M^{2} and \mu ). B.8 Telescoping and Final Rate (FedSOA with Drift) Sum ( 47 ) for r=0,\dots,R-1 and use \Delta:=F(x^{0})-F^{\star} : \displaystyle\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla F(x^{r})\|^{2} \displaystyle\leq\frac{2\Delta}{\mu\eta KR}+C_{1}\,\frac{L\eta}{\mu}\left(\frac{\sigma_{l}^{2}+\sigma_{g}^{2}}{S}+\kappa_{\Theta}\,\bar{\Delta}_{D}\right), (48) where \bar{\Delta}_{D}:=\frac{1}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r} and C_{1} is an absolute constant. Optimizing the RHS over \eta (subject to ( 46 )) yields the standard non-convex form \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla F(x^{r})\|^{2}\;\lesssim\;\mathcal{O}\!\left(\frac{L\Delta}{R}+\sqrt{\frac{L\Delta}{R}\cdot\frac{\sigma_{l}^{2}+\sigma_{g}^{2}+S\kappa_{\Theta}\,\bar{\Delta}_{D}}{SK}}\right), (49) where we use \lesssim to hide absolute constants depending only on (\mu,M) . Interpretation. Eq. 49 contains an explicit drift penalty : the larger \bar{\Delta}_{D} is, the larger the “effective noise” is, hence the slower the convergence. Appendix C Proof of Theorem 5.7 (FedPAC) C.1 Further Alignment to Theorem 5.7 (FedPAC): Why \sigma_{g}^{2} Disappears FedPAC introduces two mechanisms: (i) Preconditioner Alignment. At the beginning of round r , each participating client warm-starts from a shared reference \Theta_{i}^{r,0}\leftarrow\Theta^{r} , and the server aggregates states after local steps. This reduces preconditioner mismatch and thus controls \Delta_{D}^{r} . (ii) Local Preconditioner Correction. Each local step uses a convex combination of local preconditioned direction and an estimated global direction: x_{i}^{r,k+1}=x_{i}^{r,k}-\eta\Big[(1-\beta)\tilde{g}_{i}^{r,k}+\beta g_{G}^{r}\Big],\qquad\beta\in[0,1], (50) where g_{G}^{r} is the estimated global update from the previous round, e.g. g_{G}^{r}:=-\frac{1}{SK\eta}\sum_{i\in S_{r-1}}\big(x_{i}^{r-1,K}-x_{i}^{r-1,0}\big). (51) Algorithm 3 Federated Preconditioner Alignment and Correction Framework (FedPAC) 0: Per client, we maintain: \boldsymbol{\Theta}_{i}^{r,k} in local. Hyperparameters: learning rate \eta , communication rounds R , local updates K , the number of client N . 1: for r=0,\dots,R do 2: for each client i\in\mathcal{S}_{r} in parallel do 3: \boldsymbol{\Theta}_{i}^{r,0}\leftarrow\boldsymbol{\Theta}^{r}; 4: for k=1,\dots,K do 5: Sample batch B_{i}^{r,k} ; 6: \boldsymbol{g}_{i}^{r,k}\in\mathbb{R}^{m\times n}\leftarrow\nabla F_{i}(\boldsymbol{x}_{i}^{r,k};\xi_{i}^{r,k}) ; 7: \boldsymbol{\Theta}_{i}^{r,k+1}=\mathrm{UpdateState}\bigl(\boldsymbol{\Theta}_{i}^{r,k},\boldsymbol{g}_{i}^{r,k}\bigr); 8: \tilde{\boldsymbol{g}}_{i}^{r,k}\leftarrow\mathcal{P}_{\boldsymbol{\Theta}_{i}^{r,k}}\bigl(\boldsymbol{g}_{i}^{r,k}\bigr); 9: \boldsymbol{x}^{r,k+1}_{i}\!=\!\boldsymbol{x}^{r,k}_{i}\!\!-\!\eta_{l}[(1\!-\!\beta)\tilde{\boldsymbol{g}}_{i}^{r,k}\!+\!\beta\boldsymbol{g}_{G}^{r}] ; 10: end for 11: \Delta\boldsymbol{x}_{i}^{r}\coloneqq\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}^{r} ; 12: Client i communicate (\boldsymbol{\Delta}\boldsymbol{x}_{i}^{r},\boldsymbol{\Theta}_{i}^{r,K}) to Server; 13: end for 14: \boldsymbol{g}_{G}^{r+1}=-\frac{1}{SK\eta}\sum_{i=1}^{S}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 15: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}-\gamma\boldsymbol{g}_{G}^{r+1}; 16: \boldsymbol{\Theta}^{r+1}\;=\;\frac{1}{|\mathcal{S}_{r}|}\sum_{i\in\mathcal{S}_{r}}\boldsymbol{\Theta}_{i}^{r,K}; 17: Server broadcasts (\boldsymbol{x}^{r+1},\boldsymbol{\Theta}_{r+1},\boldsymbol{g}_{G}^{r+1}); 18: end for C.2 Further Alignment to Theorem 5.5 (FedPAC): Why \sigma_{g}^{2} Disappears (Detailed Proof) Step 0: Why \sigma_{g}^{2} appears in FedSOA. In the FedSOA proof, we explicitly insert and subtract \nabla F_{i}(x^{r}) , producing the term P_{\bar{\Theta}^{r,K}}(\nabla F_{i}(x^{r}))-P_{\bar{\Theta}^{r,K}}(\nabla F(x^{r})) . Bounding its second moment requires Assumption (A3), i.e., \|\nabla F_{i}(x)-\nabla F(x)\|^{2}\leq\sigma_{g}^{2} , which yields the \sigma_{g}^{2} term. Step 1: Use a global-centered variance assumption instead of (A3). For FedPAC, we do not introduce \nabla F_{i}(x^{r}) in the decomposition. Instead, we impose the following global-centered noise condition (which does not require bounded heterogeneity): C.3 Recursive Analysis to Eliminate Data Heterogeneity (Final) Goal. We present an appendix-ready proof template showing how to eliminate the explicit heterogeneity term (i.e., no \sigma_{g}^{2} ) via a recursive global-direction estimator, in the same spirit as DP-FedPGN-style recursion. The remaining terms depend on \sigma_{l}^{2} (variance reduced by SK ) and the preconditioner drift penalty (controlled by \Delta_{D}^{r} ). Notation. Let f(x):=\frac{1}{N}\sum_{i=1}^{N}f_{i}(x) be L -smooth. In round r , a set S_{r} of S clients participates and each performs K local steps. Let \bar{\Theta}^{r,K} be the reference preconditioner state in round r (e.g., the aggregated state), and define the local preconditioned stochastic gradient v_{i}^{r,k}:=P_{\Theta_{i}^{r,k}}\!\big(g_{i}^{r,k}\big),\qquad g_{i}^{r,k}:=\nabla f_{i}(x_{i}^{r,k};\xi_{i}^{r,k}). Define the per-round observation (average over clients and local steps) \hat{v}^{r}\;:=\;\frac{1}{SK}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}v_{i}^{r,k}. (52) Recursive global-direction estimator. We maintain a global direction estimator \{g^{r}\}_{r\geq 0} : g^{r+1}\;:=\;(1-\beta)g^{r}+\beta\hat{v}^{r},\qquad\beta\in(0,1], (53) and update the global model by x^{r+1}\;=\;x^{r}-\gamma g^{r+1}. (54) Assumptions. We use: 1. (Smoothness) f is L -smooth. 2. (Local stochastic variance) for all i,r,k , \mathbb{E}\|g_{i}^{r,k}-\nabla f_{i}(x_{i}^{r,k})\|^{2}\leq\sigma_{l}^{2} . 3. (Preconditioner boundedness) for all \Theta,v , \|P_{\Theta}(v)\|\leq M\|v\| . 4. (Preconditioner drift Lipschitz) for all \Theta,\Theta^{\prime},v , \|P_{\Theta}(v)-P_{\Theta^{\prime}}(v)\|\leq L_{\Theta}\|\Theta-\Theta^{\prime}\|\cdot\|v\| . 5. (Second-moment bound) \sup_{i,r,k}\mathbb{E}\|g_{i}^{r,k}\|^{2}\leq G^{2} . Importantly, we do not assume bounded heterogeneity \|\nabla f_{i}(x)-\nabla f(x)\|^{2}\leq\sigma_{g}^{2} . Local trajectory drift measure. Define U_{r}:=\frac{1}{SK}\sum_{i\in S_{r}}\sum_{k=0}^{K-1}\mathbb{E}\|x_{i}^{r,k}-x^{r}\|^{2}. (55) Preconditioner drift metric. Define the per-round drift \Delta_{D}^{r}:=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|\Theta_{i}^{r,K}-\bar{\Theta}^{r,K}\|^{2}. (56) Lemma C.1 (Drift-induced preconditioned disagreement) . For any vectors \{u_{i}\}_{i\in S_{r}} , \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(u_{i})-P_{\bar{\Theta}^{r,K}}(u_{i})\Big)\right\|^{2}\;\leq\;L_{\Theta}^{2}\left(\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|u_{i}\|^{2}\right)\Delta_{D}^{r}. (57) In particular, using \mathbb{E}\|u_{i}\|^{2}\leq G^{2} gives \mathbb{E}\left\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{i}^{r,k}}(u_{i})-P_{\bar{\Theta}^{r,K}}(u_{i})\Big)\right\|^{2}\;\leq\;L_{\Theta}^{2}G^{2}\,\Delta_{D}^{r}. (58) Lemma C.2 (Descent with inexact direction) . Under L -smoothness, if \gamma L\leq\frac{1}{24} , then for all r\geq 0 , \mathbb{E}\big[f(x^{r+1})\big]\leq\mathbb{E}\big[f(x^{r})\big]-\frac{11\gamma}{24}\,\mathbb{E}\|\nabla f(x^{r})\|^{2}+\frac{13\gamma}{24}\,\widetilde{\mathcal{E}}_{r}, (59) where \widetilde{\mathcal{E}}_{r}:=\mathbb{E}\|\nabla f(x^{r})-g^{r+1}\|^{2} . Proof. Since f is L -smooth, for any x,y we have f(y)\leq f(x)+\langle\nabla f(x),y-x\rangle+\frac{L}{2}\|y-x\|^{2}. Apply it with x=x^{r} and y=x^{r+1}=x^{r}-\gamma g^{r+1} : \displaystyle f(x^{r+1}) \displaystyle\leq f(x^{r})+\left\langle\nabla f(x^{r}),-\gamma g^{r+1}\right\rangle+\frac{L}{2}\gamma^{2}\|g^{r+1}\|^{2} \displaystyle=f(x^{r})-\gamma\|\nabla f(x^{r})\|^{2}+\gamma\left\langle\nabla f(x^{r}),\nabla f(x^{r})-g^{r+1}\right\rangle+\frac{L}{2}\gamma^{2}\|g^{r+1}\|^{2}. (60) For the cross term, by Young’s inequality \langle a,b\rangle\leq\frac{1}{2}\|a\|^{2}+\frac{1}{2}\|b\|^{2} , \left\langle\nabla f(x^{r}),\nabla f(x^{r})-g^{r+1}\right\rangle\leq\frac{1}{2}\|\nabla f(x^{r})\|^{2}+\frac{1}{2}\|\nabla f(x^{r})-g^{r+1}\|^{2}. (61) For the last term, use \|g^{r+1}\|^{2}=\|\nabla f(x^{r})-(\nabla f(x^{r})-g^{r+1})\|^{2}\leq 2\|\nabla f(x^{r})\|^{2}+2\|\nabla f(x^{r})-g^{r+1}\|^{2} to get \frac{L}{2}\gamma^{2}\|g^{r+1}\|^{2}\leq L\gamma^{2}\|\nabla f(x^{r})\|^{2}+L\gamma^{2}\|\nabla f(x^{r})-g^{r+1}\|^{2}. (62) Substituting ( 61 ) and ( 62 ) into ( 60 ) yields \displaystyle f(x^{r+1}) \displaystyle\leq f(x^{r})-\gamma\|\nabla f(x^{r})\|^{2}+\frac{\gamma}{2}\|\nabla f(x^{r})\|^{2}+\frac{\gamma}{2}\|\nabla f(x^{r})-g^{r+1}\|^{2} \displaystyle\quad+L\gamma^{2}\|\nabla f(x^{r})\|^{2}+L\gamma^{2}\|\nabla f(x^{r})-g^{r+1}\|^{2} \displaystyle=f(x^{r})-\Big(\frac{\gamma}{2}-L\gamma^{2}\Big)\|\nabla f(x^{r})\|^{2}+\Big(\frac{\gamma}{2}+L\gamma^{2}\Big)\|\nabla f(x^{r})-g^{r+1}\|^{2}. (63) If \gamma L\leq\frac{1}{24} , then \frac{\gamma}{2}-L\gamma^{2}=\gamma\Big(\frac{1}{2}-\gamma L\Big)\geq\gamma\Big(\frac{1}{2}-\frac{1}{24}\Big)=\frac{11\gamma}{24},\qquad\frac{\gamma}{2}+L\gamma^{2}=\gamma\Big(\frac{1}{2}+\gamma L\Big)\leq\gamma\Big(\frac{1}{2}+\frac{1}{24}\Big)=\frac{13\gamma}{24}. Thus ( 63 ) implies f(x^{r+1})\leq f(x^{r})-\frac{11\gamma}{24}\|\nabla f(x^{r})\|^{2}+\frac{13\gamma}{24}\|\nabla f(x^{r})-g^{r+1}\|^{2}. Taking expectation on both sides and recalling \widetilde{\mathcal{E}}_{r}:=\mathbb{E}\|\nabla f(x^{r})-g^{r+1}\|^{2} completes the proof. ∎ Lemma C.3 (Recursive estimator error (no explicit \sigma_{g}^{2} )) . Let \widetilde{\mathcal{E}}_{r}:=\mathbb{E}\|\nabla f(x^{r})-g^{r+1}\|^{2} with g^{r+1} defined in ( 53 )–( 52 ). If \gamma L\leq\frac{\beta}{6} , then for all r\geq 1 , \widetilde{\mathcal{E}}_{r}\;\leq\;\Big(1-\frac{8\beta}{9}\Big)\widetilde{\mathcal{E}}_{r-1}+\frac{C_{1}\gamma^{2}L^{2}}{\beta}\,\mathbb{E}\|\nabla f(x^{r-1})\|^{2}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{r}+C_{4}\beta\,L_{\Theta}^{2}G^{2}\,\Delta_{D}^{r}, (64) and for r=0 , \widetilde{\mathcal{E}}_{0}\;\leq\;(1-\beta)\widetilde{\mathcal{E}}_{-1}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{0}+C_{4}\beta\,L_{\Theta}^{2}G^{2}\,\Delta_{D}^{0}, (65) where C_{1},C_{2},C_{3},C_{4}>0 are absolute constants depending only on (M) . Proof. We follow a DP-FedPGN-style recursion and avoid introducing \nabla f_{i}-\nabla f explicitly. Step 1: Expand the recursion. Define \delta_{r}:=\nabla f(x^{r})-g^{r+1} so that \widetilde{\mathcal{E}}_{r}=\mathbb{E}\|\delta_{r}\|^{2} . From ( 53 ) and ( 52 ), \displaystyle\delta_{r} \displaystyle=\nabla f(x^{r})-(1-\beta)g^{r}-\beta\hat{v}^{r} \displaystyle=(1-\beta)\big(\nabla f(x^{r})-g^{r}\big)+\beta\big(\nabla f(x^{r})-\hat{v}^{r}\big). (66) Add and subtract \nabla f(x^{r-1}) inside the first bracket: \displaystyle\nabla f(x^{r})-g^{r} \displaystyle=\underbrace{\nabla f(x^{r})-\nabla f(x^{r-1})}_{A_{r}}+\underbrace{\nabla f(x^{r-1})-g^{r}}_{=\delta_{r-1}}. (67) Plugging ( 67 ) into ( 66 ) gives \delta_{r}=(1-\beta)\delta_{r-1}+(1-\beta)A_{r}+\beta B_{r},\qquad B_{r}:=\nabla f(x^{r})-\hat{v}^{r}. (68) Step 2: Square and apply the same AM-GM pattern. Using \|a+b\|^{2}\leq(1+\frac{\beta}{2})\|a\|^{2}+(1+\frac{2}{\beta})\|b\|^{2} twice (as in your proof), one obtains \mathbb{E}\|\delta_{r}\|^{2}\leq\Big(1+\frac{\beta}{2}\Big)(1-\beta)^{2}\mathbb{E}\|\delta_{r-1}\|^{2}+\frac{C}{\beta}\,\mathbb{E}\|A_{r}\|^{2}+C\beta^{2}\,\mathbb{E}\|B_{r}\|^{2}, (69) for an absolute constant C>0 . Step 3: Bound \mathbb{E}\|A_{r}\|^{2} by smoothness and the update. By L -smoothness, \|\nabla f(x^{r})-\nabla f(x^{r-1})\|\leq L\|x^{r}-x^{r-1}\| and x^{r}-x^{r-1}=-\gamma g^{r} , hence \displaystyle\mathbb{E}\|A_{r}\|^{2} \displaystyle\leq L^{2}\gamma^{2}\,\mathbb{E}\|g^{r}\|^{2}\leq 2L^{2}\gamma^{2}\Big(\mathbb{E}\|\nabla f(x^{r-1})\|^{2}+\mathbb{E}\|\nabla f(x^{r-1})-g^{r}\|^{2}\Big)=2L^{2}\gamma^{2}\Big(\mathbb{E}\|\nabla f(x^{r-1})\|^{2}+\widetilde{\mathcal{E}}_{r-1}\Big). (70) Step 4: Bound \mathbb{E}\|B_{r}\|^{2} without \sigma_{g}^{2} . Recall B_{r}=\nabla f(x^{r})-\hat{v}^{r} and \hat{v}^{r}=\frac{1}{SK}\sum_{i,k}P_{\Theta_{i}^{r,k}}(g_{i}^{r,k}) . Add and subtract the reference-preconditioned terms: \displaystyle B_{r} \displaystyle=\underbrace{\nabla f(x^{r})-\frac{1}{SK}\sum_{i,k}P_{\bar{\Theta}^{r,K}}\!\big(g_{i}^{r,k}\big)}_{B_{r}^{(1)}}+\underbrace{\frac{1}{SK}\sum_{i,k}\Big(P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})-P_{\Theta_{i}^{r,k}}(g_{i}^{r,k})\Big)}_{B_{r}^{(2)}}. (71) (i) Stochastic + trajectory part B_{r}^{(1)} . Insert and subtract \nabla f(x_{i}^{r,k}) and use \|P_{\bar{\Theta}}(v)\|\leq M\|v\| : \displaystyle\|B_{r}^{(1)}\|^{2} \displaystyle=\left\|\frac{1}{SK}\sum_{i,k}\Big(\nabla f(x^{r})-P_{\bar{\Theta}^{r,K}}(g_{i}^{r,k})\Big)\right\|^{2} \displaystyle\leq\frac{2}{SK}\sum_{i,k}\left\|\nabla f(x^{r})-P_{\bar{\Theta}^{r,K}}(\nabla f(x_{i}^{r,k}))\right\|^{2}+\frac{2}{SK}\sum_{i,k}\left\|P_{\bar{\Theta}^{r,K}}(\nabla f(x_{i}^{r,k})-g_{i}^{r,k})\right\|^{2} \displaystyle\leq\frac{2M^{2}}{SK}\sum_{i,k}\|\nabla f(x^{r})-\nabla f(x_{i}^{r,k})\|^{2}+\frac{2M^{2}}{SK}\sum_{i,k}\|\nabla f(x_{i}^{r,k})-g_{i}^{r,k}\|^{2}. (72) Taking expectation, using local variance \mathbb{E}\|g_{i}^{r,k}-\nabla f_{i}(x_{i}^{r,k})\|^{2}\leq\sigma_{l}^{2} and L -smoothness \|\nabla f(x_{i}^{r,k})-\nabla f(x^{r})\|^{2}\leq L^{2}\|x_{i}^{r,k}-x^{r}\|^{2} , and absorbing the client/gradient mismatch into the recursion (rather than a \sigma_{g}^{2} constant), we get the standard bound \mathbb{E}\|B_{r}^{(1)}\|^{2}\leq C\left(\frac{\sigma_{l}^{2}}{SK}+L^{2}U_{r}\right), (73) where U_{r} is defined in ( 55 ) and C>0 depends only on M . (ii) Preconditioner drift part B_{r}^{(2)} . By Lemma C.1 with u_{i}=g_{i}^{r,k} and ( 58 ), \mathbb{E}\|B_{r}^{(2)}\|^{2}\leq C\,L_{\Theta}^{2}G^{2}\,\Delta_{D}^{r}. (74) Combining ( 71 )–( 74 ) gives \mathbb{E}\|B_{r}\|^{2}\leq C\left(\frac{\sigma_{l}^{2}}{SK}+L^{2}U_{r}+L_{\Theta}^{2}G^{2}\,\Delta_{D}^{r}\right). (75) Step 5: Conclude the recursion and absorb \widetilde{\mathcal{E}}_{r-1} . Plug ( 70 ) and ( 75 ) into ( 69 ). Using \gamma L\leq\beta/6 to absorb the \widetilde{\mathcal{E}}_{r-1} term into the contraction, we obtain ( 64 )–( 65 ) with absolute constants. ∎ Theorem C.4 (Non-convex convergence without explicit heterogeneity) . Under the assumptions above, take g^{0}=0 and choose \beta\asymp\min\left\{1,\sqrt{\frac{SKL\Delta}{\sigma_{l}^{2}R}}\right\},\qquad\gamma=\min\left\{\frac{1}{24L},\frac{\beta}{6L}\right\}, and \eta KL sufficiently small so that the local drift bound U_{r} is controlled (as in the auxiliary lemma). Then we have \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}\;\lesssim\;\frac{L\Delta}{R}+\sqrt{\frac{L\Delta\,\sigma_{l}^{2}}{SKR}}\;+\;\underbrace{\gamma L_{\Theta}^{2}G^{2}\cdot\frac{1}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r}}_{\text{drift penalty}}. (76) In particular, under alignment \Delta_{D}^{r}\approx 0 , the last term vanishes and the rate depends only on \sigma_{l}^{2} . Proof. Summing the descent inequality in Lemma C.2 over r=0,\dots,R-1 yields \frac{11\gamma}{24}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}\leq\Delta+\frac{13\gamma}{24}\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r},\qquad\Delta:=f(x^{0})-f^{\star}. (77) It remains to bound \sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r} . Step 1: Sum the recursion for \widetilde{\mathcal{E}}_{r} . From Lemma C.3 , for r\geq 1 , \widetilde{\mathcal{E}}_{r}\leq\Big(1-\frac{8\beta}{9}\Big)\widetilde{\mathcal{E}}_{r-1}+\frac{C_{1}\gamma^{2}L^{2}}{\beta}\,\mathbb{E}\|\nabla f(x^{r-1})\|^{2}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{r}+C_{4}\beta\,L_{\Theta}^{2}G^{2}\,\Delta_{D}^{r}. Summing both sides over r=1,\dots,R-1 gives \displaystyle\sum_{r=1}^{R-1}\widetilde{\mathcal{E}}_{r} \displaystyle\leq\Big(1-\frac{8\beta}{9}\Big)\sum_{r=1}^{R-1}\widetilde{\mathcal{E}}_{r-1}+\frac{C_{1}\gamma^{2}L^{2}}{\beta}\sum_{r=1}^{R-1}\mathbb{E}\|\nabla f(x^{r-1})\|^{2}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}(R-1) \displaystyle\quad+C_{3}\beta L^{2}\sum_{r=1}^{R-1}U_{r}+C_{4}\beta L_{\Theta}^{2}G^{2}\sum_{r=1}^{R-1}\Delta_{D}^{r}. (78) Re-index \sum_{r=1}^{R-1}\widetilde{\mathcal{E}}_{r-1}=\sum_{r=0}^{R-2}\widetilde{\mathcal{E}}_{r} and \sum_{r=1}^{R-1}\mathbb{E}\|\nabla f(x^{r-1})\|^{2}=\sum_{r=0}^{R-2}\mathbb{E}\|\nabla f(x^{r})\|^{2} . Also note \sum_{r=1}^{R-1}\widetilde{\mathcal{E}}_{r}\leq\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r} and \sum_{r=0}^{R-2}\widetilde{\mathcal{E}}_{r}\leq\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r} . Thus ( 78 ) implies \displaystyle\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r} \displaystyle\leq\Big(1-\frac{8\beta}{9}\Big)\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r}+\widetilde{\mathcal{E}}_{0}+\frac{C_{1}\gamma^{2}L^{2}}{\beta}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}R \displaystyle\quad+C_{3}\beta L^{2}\sum_{r=0}^{R-1}U_{r}+C_{4}\beta L_{\Theta}^{2}G^{2}\sum_{r=0}^{R-1}\Delta_{D}^{r}. (79) Move the contraction term to the left: \frac{8\beta}{9}\sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r}\leq\widetilde{\mathcal{E}}_{0}+\frac{C_{1}\gamma^{2}L^{2}}{\beta}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}R+C_{3}\beta L^{2}\sum_{r=0}^{R-1}U_{r}+C_{4}\beta L_{\Theta}^{2}G^{2}\sum_{r=0}^{R-1}\Delta_{D}^{r}. (80) Therefore, \sum_{r=0}^{R-1}\widetilde{\mathcal{E}}_{r}\leq\frac{9}{8\beta}\widetilde{\mathcal{E}}_{0}+\frac{9C_{1}}{8}\frac{\gamma^{2}L^{2}}{\beta^{2}}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}+\frac{9C_{2}}{8}\frac{\beta\sigma_{l}^{2}}{SK}R+\frac{9C_{3}}{8}L^{2}\sum_{r=0}^{R-1}U_{r}+\frac{9C_{4}}{8}L_{\Theta}^{2}G^{2}\sum_{r=0}^{R-1}\Delta_{D}^{r}. (81) Step 2: Control \widetilde{\mathcal{E}}_{0} by \widetilde{\mathcal{E}}_{-1} . From Lemma C.3 at r=0 , \widetilde{\mathcal{E}}_{0}\leq(1-\beta)\widetilde{\mathcal{E}}_{-1}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{0}+C_{4}\beta L_{\Theta}^{2}G^{2}\Delta_{D}^{0}\leq\widetilde{\mathcal{E}}_{-1}+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{0}+C_{4}\beta L_{\Theta}^{2}G^{2}\Delta_{D}^{0}. If g^{0}=0 , then \widetilde{\mathcal{E}}_{-1}=\mathbb{E}\|\nabla f(x^{0})-g^{0}\|^{2}=\|\nabla f(x^{0})\|^{2}\leq 2L\big(f(x^{0})-f^{\star}\big)=2L\Delta (by standard smoothness inequality). Hence \widetilde{\mathcal{E}}_{0}\;\leq\;2L\Delta+\frac{C_{2}\beta^{2}\sigma_{l}^{2}}{SK}+C_{3}\beta L^{2}U_{0}+C_{4}\beta L_{\Theta}^{2}G^{2}\Delta_{D}^{0}. (82) Step 3: Plug ( 81 ) into ( 77 ) and isolate \sum\|\nabla f(x^{r})\|^{2} . Substitute ( 81 ) into ( 77 ): \displaystyle\frac{11\gamma}{24}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2} \displaystyle\leq\Delta+\frac{13\gamma}{24}\Bigg[\frac{9}{8\beta}\widetilde{\mathcal{E}}_{0}+\frac{9C_{1}}{8}\frac{\gamma^{2}L^{2}}{\beta^{2}}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}+\frac{9C_{2}}{8}\frac{\beta\sigma_{l}^{2}}{SK}R \displaystyle\qquad\qquad\qquad+\frac{9C_{3}}{8}L^{2}\sum_{r=0}^{R-1}U_{r}+\frac{9C_{4}}{8}L_{\Theta}^{2}G^{2}\sum_{r=0}^{R-1}\Delta_{D}^{r}\Bigg]. (83) Move the term containing \sum\mathbb{E}\|\nabla f(x^{r})\|^{2} on the RHS to the LHS. Specifically, assume \gamma L\leq\beta/6 so that \frac{13\gamma}{24}\cdot\frac{9C_{1}}{8}\cdot\frac{\gamma^{2}L^{2}}{\beta^{2}}\;\leq\;\frac{1}{2}\cdot\frac{11\gamma}{24}, (84) which can always be ensured by taking \gamma L\lesssim\beta (as in the theorem statement). Then ( 83 ) implies \displaystyle\frac{11\gamma}{48}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2} \displaystyle\leq\Delta+\frac{13\gamma}{24}\cdot\frac{9}{8\beta}\widetilde{\mathcal{E}}_{0}+\frac{13\gamma}{24}\cdot\frac{9C_{2}}{8}\frac{\beta\sigma_{l}^{2}}{SK}R \displaystyle\quad+\frac{13\gamma}{24}\cdot\frac{9C_{3}}{8}L^{2}\sum_{r=0}^{R-1}U_{r}+\frac{13\gamma}{24}\cdot\frac{9C_{4}}{8}L_{\Theta}^{2}G^{2}\sum_{r=0}^{R-1}\Delta_{D}^{r}. (85) Divide both sides by R and by \gamma : \displaystyle\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2} \displaystyle\leq C\left(\frac{\Delta}{\gamma R}+\frac{\widetilde{\mathcal{E}}_{0}}{\beta R}+\frac{\beta\sigma_{l}^{2}}{SK}+\frac{L^{2}}{R}\sum_{r=0}^{R-1}U_{r}+\frac{L_{\Theta}^{2}G^{2}}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r}\right), (86) where C>0 is an absolute constant. Step 4: Remove \widetilde{\mathcal{E}}_{0} and handle U_{r} (auxiliary bound). Plug ( 82 ) into ( 86 ): \displaystyle\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2} \displaystyle\leq C\left(\frac{\Delta}{\gamma R}+\frac{L\Delta}{\beta R}+\frac{\beta\sigma_{l}^{2}}{SK}+\frac{L^{2}}{R}\sum_{r=0}^{R-1}U_{r}+\frac{L_{\Theta}^{2}G^{2}}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r}\right)+C\cdot\frac{\beta\sigma_{l}^{2}}{SK}\cdot\frac{1}{R}, (87) where the last tiny term can be absorbed into \beta\sigma_{l}^{2}/(SK) . Next, apply the auxiliary bound on U_{r} (as in your reference proof) to ensure \frac{L^{2}}{R}\sum_{r=0}^{R-1}U_{r}\;\leq\;C^{\prime}\frac{\beta\sigma_{l}^{2}}{SK}, (88) by choosing \eta KL sufficiently small. This yields \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}\;\leq\;C\left(\frac{\Delta}{\gamma R}+\frac{L\Delta}{\beta R}+\frac{\beta\sigma_{l}^{2}}{SK}+\frac{L_{\Theta}^{2}G^{2}}{R}\sum_{r=0}^{R-1}\Delta_{D}^{r}\right). (89) Step 5: Choose (\beta,\gamma) and simplify to the final rate. Choose \gamma=\min\{\frac{1}{24L},\frac{\beta}{6L}\} and \beta\asymp\min\left\{1,\sqrt{\frac{SKL\Delta}{\sigma_{l}^{2}R}}\right\}. Then \Delta/(\gamma R)\lesssim L\Delta/R and \frac{L\Delta}{\beta R}+\frac{\beta\sigma_{l}^{2}}{SK}\;\lesssim\;\sqrt{\frac{L\Delta\sigma_{l}^{2}}{SKR}}. Plugging these into ( 89 ) yields \frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^{r})\|^{2}\;\lesssim\;\frac{L\Delta}{R}+\sqrt{\frac{L\Delta\,\sigma_{l}^{2}+\kappa_{\Theta}\cdot\bar{\Delta}_{D}}{SKR}}, which is exactly ( 76 ). ∎ Appendix D Appendix B: Experimental Setup D.1 Setting for ResNet-18 Table 7 : A detailed summary of 100 and Tiny-ImageNet: number of classes, image size, and dataset splits. Dataset #Classes Image Size Train Val Test Total Train / class CIFAR-100 100 3\times 32\times 32 50,000 — 10,000 60,000 500 Tiny ImageNet 200 3\times 64\times 64 100,000 10,000 10,000 120,000 500 Notes. (1) CIFAR-10/100 provide no official validation split; a subset of the training set is commonly reserved as dev/val. (2) CIFAR-100 contains 100 fine-grained classes; 20 coarse superclasses are also defined for hierarchical labeling. (3) Tiny ImageNet is a subset of ImageNet synsets: per class 500 train, 50 val, and 50 test images (test labels are not publicly released). (4) All three datasets are single-label classification with RGB images resized to fixed resolutions. We evaluate our methods on two widely-used benchmark datasets in federated learning: CIFAR-100 and Tiny ImageNet . • CIFAR-100 ( 12 ) : Contains 100 classes with 600 color images per class at a resolution of 32\times 32 . It is a standard benchmark for evaluating federated image classification methods. • Tiny ImageNet : A subset of ImageNet with 200 classes and 500 images per class, providing a more challenging and high-resolution classification task. D.2 Federated Learning Configuration We simulate a cross-device federated learning environment using the following settings: Table 8 : Hyperparameter configuration of ResNet-18 and Vit-Tiny (CIFAR100, Tiny-ImageNet) across different algorithms. Method Local Optimizer Local LR \beta \beta_{1} \beta_{2} Weight Decay FedAvg (Local SGD) SGD 0.1 — — — 0.001 SCAFFOLD SGD 0.1 — — — 0.001 FedCM SGD 0.1 0.9 — — 0.001 Local AdamW AdamW 3e-4 — 0.9 0.999 0.01 Local Sophia Sophia 3e-4 — 0.9 0.99 0.01 Local Muon Muon 3e-2 — 0.9 0.95 0.01 Local SOAP SOAP 3e-3 — 0.95 0.95 0.01 FedPAC_Sophia Muon 3e-4 0.5 0.9 0.99 0.01 FedPAC_Muon Muon 3e-2 0.5 0.9 0.95 0.01 FedPAC_SOAP SOAP 3e-3 0.5 0.95 0.95 0.01 Table 9 : Hyperparameter configuration of ViT-Base fine-tuning across different algorithms. Method Local Optimizer Local LR \beta \beta_{1} \beta_{2} Weight Decay FedAvg (Local SGD) SGD 0.1 — — — 0.001 SCAFFOLD SGD 0.1 — — — 0.001 FedCM SGD 0.1 0.9 — — 0.001 Local AdamW AdamW 1e-4 — 0.9 0.999 0.01 Local Sophia Sophia 1e-4 — 0.9 0.99 0.01 Local Muon Muon 1e-2 — 0.9 0.95 0.01 Local SOAP SOAP 1e-3 — 0.95 0.95 0.01 FedPAC_Sophia Muon 1e-4 0.5 0.9 0.99 0.01 FedPAC_Muon Muon 1e-2 0.5 0.9 0.95 0.01 FedPAC_SOAP SOAP 1e-3 0.5 0.95 0.95 0.01 D.3 Model Architecture We adopt ResNet-18 as the backbone model. To better adapt it to CIFAR-100, we modify its architecture following standard practices ( 7 ) : • Replace the original 7\times 7 convolution with a 3\times 3 kernel. • Remove the initial downsampling layers (stride-2 convolution and max pooling). We also compare Batch Normalization (BN) and Group Normalization (GN) in ResNet-18. Empirically, BN outperforms GN on CIFAR-100, so we adopt the BN-based version, denoted as ResNet-18-BN , throughout our experiments. D.4 Setting for ViT-Tiny We construct a lightweight Vision Transformer model, ViT-Tiny , specifically tailored for federated learning on the CIFAR-100 dataset. The design is based on the standard ViT architecture ( 4 ) , with modifications to accommodate the small input size and limited data per client. • Input resolution: 32\times 32 • Patch size: 4\times 4 , resulting in 64 tokens per image • Embedding dimension: 192 • Number of Transformer layers: 6 • Number of attention heads: 3 • Normalization: LayerNorm (applied before attention and MLP blocks) • Classification head: Linear projection to 100 classes (CIFAR-100) • Activation: GELU • Initialization: Xavier/Glorot for linear layers; sinusoidal positional encoding To regularize training, we apply dropout (0.1) to both attention and MLP layers. All models are trained from scratch without pretraining. Remarks. Due to the smaller capacity of ViT-Tiny and limited data per client, we find that careful normalization (e.g., LayerNorm placement) and early learning rate warmup are beneficial. For future work, more advanced token-mixing techniques or hybrid CNN-ViT backbones may further improve performance in federated settings. D.5 Transformer Fine-tuning Settings To demonstrate the effectiveness of our method on large-scale vision models, we conduct fine-tuning experiments using ViT-Base on Tiny ImageNet and CIFAR-100 . For both models, we initialize from official ImageNet-22K pre-trained weights ( 18 ; 4 ) to ensure consistency across methods. We fine-tune all layers during federated training. Data Preprocessing. To align with the input resolution required by ViT, we resize images from both datasets to 224\times 224 using bilinear interpolation. Standard data augmentation techniques such as random cropping, horizontal flipping, and RandAugment are applied locally at the client side. D.6 Additional Federated Training Configuration of LLM To evaluate our algorithm under a smaller-scale federation, we further conduct experiments with a reduced number of clients and adjusted participation parameters. Federated Setup. We simulate a federated learning environment with the following configuration: Table 10 : Hyperparameter configuration of LLAMA (C4) across different algorithms. Method Local Optimizer Local LR \beta \beta_{1} \beta_{2} Weight Decay FedAvg (Local SGD) SGD 0.1 — — — 0.001 SCAFFOLD SGD 0.1 — — — 0.001 FedCM SGD 0.1 0.9 — — 0.001 Local AdamW AdamW 3e-4 — 0.9 0.999 0.01 Local Sophia Sophia 3e-4 — 0.9 0.99 0.01 Local Muon Muon 3e-2 — 0.9 0.95 0.01 Local SOAP SOAP 3e-3 — 0.95 0.95 0.01 FedPAC_Sophia Muon 3e-4 0.5 0.9 0.99 0.01 FedPAC_Muon Muon 3e-2 0.5 0.9 0.95 0.01 FedPAC_SOAP SOAP 3e-3 0.5 0.95 0.95 0.01 Appendix E Appendix C: Experimental Appendix E.1 Communication and Computation Cost Analysis Table 11 : Per-round communication cost of different momentum aggregation strategies. Here |x| denotes the number of model parameters (in floats), and \operatorname{CommCost} is per-round communication cost, Compute-Cost is per-round computation time. (ViT-Tiny, R=300 , Dir-0.1, K=50 ) Method / Strategy Communication CommCost Compute-Cost (s) Acc(%) FedAvg |x| 22.8 MB 4.56 s 27.24 SCAFFOLD 2|x| 45.6 MB 5.22 s 26.86 FedCM |x| 22.8 MB 4.68 s 16.95 Local AdamW |x| 22.8 MB 4.89 s 37.57 Local Sophia |x| 22.8 MB 4.92 s 34.05 Local Muon |x| 22.8 MB 5.14 s 44.00 Local SOAP |x| 22.8 MB 5.56 s 49.41 FedPAC_Sophia |x|+|\Theta| 45.6 MB 5.08 s 39.79 FedPAC_Muon |x|+1|\Theta| 45.6 MB 5.25 s 47.81 FedPAC_SOAP |x|+2|\Theta| 68.4 MB 5.68 s 51.16 FedPAC_Sophia_Light |x|+0.05|\Theta| 23.9 MB 5.11 s 39.45 FedPAC_Muon_Light |x|+0.05|\Theta| 23.9 MB 5.28 s 47.23 FedPAC_SOAP_Light |x|+0.1|\Theta| 25.2 MB 5.69 s 50.56 Table 12 : Test accuracy, training loss of each method on CIFAR-100 and Tiny-Imagenet using ResNet-18 over 300 communication rounds under IID Dir-0.5, Dir-0.1, Dir-0.05 (100 clients, 10% participation, batch size 50, K=50 ). Method CIFAR-100 (ResNet-18) Tiny-Imagenet (ResNet-18) iid Dir-0.5 Dir-0.1 Dir-0.05 iid Dir-0.5 Dir-0.1 Dir-0.05 FedAvg 65.04 65.07 60.17 56.75 53.88 52.55 47.48 43.80 SCAFFOLD 66.04 65.51 60.69 56.43 54.49 53.36 47.76 43.92 FedCM 71.12 69.85 66.61 62.65 46.51 44.06 41.16 36.00 Local AdamW 64.37 63.58 59.23 55.24 51.06 49.97 44.01 40.00 Fed-Sophia 62.56 60.62 57.29 51.02 49.86 47.62 41.89 36.65 FedPM 62.86 61.23 57.25 50.98 49.63 48.21 41.53 36.62 Local Sophia 62.17 60.96 56.65 50.89 49.28 47.99 41.23 36.15 Local Muon 72.78 \mathbf{73.02} 67.26 49.86 \mathbf{60.50} \mathbf{60.40} 52.83 34.76 Local SOAP 71.98 71.19 68.44 58.16 58.01 56.82 54.42 50.02 FedPAC_Sophia 64.90 64.71 59.96 53.66 49.30 50.92 43.81 36.37 FedPAC_Muon \mathbf{72.79} 72.50 \mathbf{71.85} \mathbf{65.56} 58.85 58.31 \mathbf{57.95} \mathbf{54.00} FedPAC_SOAP 69.79 71.05 69.25 64.16 56.57 56.22 55.62 51.81 Table 13 : Test accuracy, training loss of each method on CIFAR-100 using ViT-Tiny over 300 communication rounds under IID, Dir-0.5, Dir-0.1, Dir-0.05 (100 clients, 10% participation, batch size 50, K=50 ). Method CIFAR-100 (ViT-Tiny) Tiny-Imagenet (ViT-Tiny) iid Dir-0.5 Dir-0.1 Dir-0.05 iid Dir-0.5 Dir-0.1 Dir-0.05 FedAvg 32.97 33.66 27.24 23.42 18.61 18.03 15.68 14.05 SCAFFOLD 32.29 33.01 26.86 23.23 18.32 18.36 15.70 14.21 FedCM 21.97 22.02 16.95 14.74 10.23 11.66 8.88 8.15 Local AdamW 41.27 41.21 37.57 36.06 27.94 26.82 24.31 21.35 Fed-Sophia 38.34 38.56 34.23 32.27 23.56 24.46 23.85 22.17 FedPM 38.56.94 38.75 34.29 32.62 23.86 24.51 23.89 22.85 Local Sophia 37.94 38.48 34.05 32.25 22.90 23.46 22.49 21.14 Local Muon 50.50 51.75 47.81 41.76 34.80 34.57 30.51 28.25 Local SOAP 52.85 51.61 49.41 41.68 34.81 33.91 33.30 30.36 FedPAC_Sophia 42.53 42.29 39.79 32.71 23.26 22.16 23.01 22.37 FedPAC_Muon 52.70 52.02 44.00 39.68 31.78 33.48 31.45 30.25 FedPAC_SOAP 52.86 52.46 51.16 47.55 35.56 35.12 34.32 31.33 E.2 More baseline experiment comparisons Training on CIFAR-100 with ResNet-18. Table 12 reports the final test accuracy after 300 communication rounds on CIFAR-100 and Tiny-ImageNet with ResNet-18 under increasing data heterogeneity (IID, Dir- 0.5 , Dir- 0.1 , Dir- 0.05 ; 100 clients, 10% participation, batch size 50, and K{=}50 local steps). Overall, performance degrades as the Dirichlet concentration decreases, highlighting the non-trivial impact of client drift under severe non-IID distributions. Specialized second-order FL baselines. To directly address the concern that our gains may come from comparing against local second-order optimizers only, we additionally include two specialized second-order FL methods, Fed-Sophia and FedPM , which are designed to adapt curvature-aware (Sophia-style) updates to the federated setting. As shown in Table 12 , both specialized baselines still exhibit noticeable performance drops as heterogeneity increases. For example, on CIFAR-100, Fed-Sophia decreases from 60.62 (Dir- 0.5 ) to 51.02 (Dir- 0.05 ), and FedPM decreases from 61.23 (Dir- 0.5 ) to 50.98 (Dir- 0.05 ), indicating that merely introducing a specialized second-order aggregation scheme does not fully resolve the instability induced by preconditioner drift . FedPAC improves robustness under strong heterogeneity. Across both datasets, FedPAC_Muon yields the best or near-best accuracy under strong non-IID settings, demonstrating substantial robustness benefits. On CIFAR-100 with Dir- 0.05 , FedPAC_Muon achieves 65.56, improving over FedAvg (56.75) by +8.81 and over Local Muon (49.86) by +15.70. On Tiny-ImageNet with Dir- 0.05 , FedPAC_Muon reaches 54.00, outperforming FedAvg (43.80) by +10.20 and Local Muon (34.76) by +19.24. These results align with our analysis that the server-side alignment and client-side correction in FedPAC explicitly mitigate the mismatch among client preconditioners, which becomes particularly harmful when data heterogeneity is severe. FedPAC also complements Sophia-style methods. Compared with the specialized Sophia-style FL baselines, FedPAC_Sophia provides consistent gains on CIFAR-100 across all heterogeneity levels (e.g., 53.66 vs. 51.02/50.98 under Dir- 0.05 for Fed-Sophia/FedPM). On Tiny-ImageNet, FedPAC_Sophia is competitive with Fed-Sophia/FedPM and improves them under mild-to-moderate heterogeneity (IID/Dir- 0.5 /Dir- 0.1 ), while being essentially on par under Dir- 0.05 . Taken together, these comparisons suggest that FedPAC addresses a more general failure mode— preconditioner drift —and thus can serve as a robust, optimizer-agnostic enhancement for federated second-order training. Takeaway. Table 12 verifies that (i) directly deploying local second-order optimizers can be brittle under strong heterogeneity, and (ii) even specialized second-order FL methods may not fully eliminate the degradation, whereas FedPAC substantially improves robustness and accuracy, especially in the most heterogeneous regimes. Training on CIFAR-100 with ViT-Tiny. E.3 More baseline experiment on iid data To further strengthen the empirical comparisons (especially on IID data) and to examine the robustness trend as heterogeneity increases, we report additional results under IID and Dirichlet partitions with varying concentration parameters. Tables 12 and 13 summarize the final test accuracy after 300 communication rounds on CIFAR-100 and Tiny-ImageNet with ResNet-18 and ViT-Tiny , respectively (100 clients, 10% participation, batch size 50, K{=}50 ). ResNet-18 results. From Table 12 , we observe that under IID (and mild heterogeneity such as Dir- 0.5 ), several strong baselines can be competitive (e.g., Local Muon on CIFAR-100). However, as heterogeneity becomes severe (Dir- 0.1 and Dir- 0.05 ), methods that rely on purely local second-order states (e.g., Local Muon ) or specialized second-order FL baselines (e.g., Fed-Sophia and FedPM ) degrade markedly, indicating the growing impact of preconditioner drift across clients. In contrast, FedPAC_Muon remains consistently strong and achieves the best accuracy under the most heterogeneous regime on both datasets, demonstrating improved robustness under non-IID distributions. ViT-Tiny results. Table 13 shows a similar trend for transformer-style backbones: while several baselines perform reasonably under IID/Dir- 0.5 , severe heterogeneity leads to substantial drops. Notably, FedPAC_SOAP achieves the best performance across all heterogeneity levels on both CIFAR-100 and Tiny-ImageNet with ViT-Tiny, suggesting that FedPAC can effectively stabilize curvature-aware training for modern architectures under client heterogeneity. Appendix F Related Work Family What is synchronized (server \leftrightarrow clients)? Extra cost (comm./state) Main issue it targets FedAvg / Local-SGD Model parameters/updates x only \approx|x| per round General FL baseline; does not explicitly correct non-IID induced drift FedOpt (server adaptive; e.g., FedAdam/FedYogi) x only; optimizer state (moments) kept on server (not shared as a common client geometry) \approx|x| (no extra client-side sync) Improves global update scaling/adaptivity, but does not align heterogeneous client metrics Control-variate / drift correction (e.g., SCAFFOLD) x + first-order control variates (gradient-level states) typically larger (e.g., SCAFFOLD uses \approx 2|x| in our comm accounting) Corrects first-order client drift under non-IID via control variates; operates at gradient level, not metric level Naive second-order FL (Local Sophia/SOAP/Muon + FedAvg agg.) x only; each client maintains local preconditioner state \Theta but does not synchronize it \approx|x| per round Fails under non-IID due to preconditioner drift (mismatched curvature-induced geometries) FedPAC (ours) x + preconditioner state \Theta (geometry / metric) : server aggregates \Theta and broadcasts a global reference (Alignment); local steps are corrected using a global preconditioned direction (Correction) \approx|x|+c|\Theta| (e.g., c=1 for Muon/Sophia, c=2 for SOAP; light variants use low-rank upload such as 0.05|\Theta| – 0.1|\Theta| ) Targets preconditioner drift directly by synchronizing the metric (Alignment) and suppressing long-term drift (Correction) Table 14: Positioning of FedPAC. Unlike FedOpt and control-variate methods that synchronize only parameter/first-order states, FedPAC explicitly synchronizes the curvature-defined geometry (preconditioner state \Theta ) to mitigate preconditioner drift under non-IID data. Here |x| denotes per-round model communication in our accounting and |\Theta| denotes the size of the optimizer preconditioner state. Proposition F.1 (Geometry drift induces preconditioned disagreement) . Consider a second-order FL update where each client applies a preconditioner operator P_{\Theta} (defined by its optimizer state \Theta ) to a local vector u_{i} (e.g., stochastic gradient). Assume P_{\Theta} is Lipschitz in the state: for any \Theta,\Theta^{\prime} and any v , \|P_{\Theta}(v)-P_{\Theta^{\prime}}(v)\|\leq L_{\Theta}\,\|\Theta-\Theta^{\prime}\|\,\|v\|. (90) Define the per-round preconditioner drift metric \Delta_{r}^{D}:=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|\Theta_{r,K}^{i}-\bar{\Theta}_{r,K}\|^{2}. (91) Then for any collection \{u_{i}\}_{i\in S_{r}} , \mathbb{E}\Big\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{r,k}^{i}}(u_{i})-P_{\bar{\Theta}_{r,K}}(u_{i})\Big)\Big\|^{2}\;\leq\;L_{\Theta}^{2}\Big(\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|u_{i}\|^{2}\Big)\Delta_{r}^{D}. (92) In particular, if \sup_{i,r,k}\mathbb{E}\|u_{i}\|^{2}\leq G^{2} , then \mathbb{E}\Big\|\frac{1}{S}\sum_{i\in S_{r}}\Big(P_{\Theta_{r,k}^{i}}(u_{i})-P_{\bar{\Theta}_{r,K}}(u_{i})\Big)\Big\|^{2}\;\leq\;L_{\Theta}^{2}G^{2}\Delta_{r}^{D}. (93) Therefore, even when there is no gradient/client mismatch (e.g., u_{i}\equiv u for all clients, or heterogeneity is negligible), heterogeneous optimizer states \Theta_{r,k}^{i} alone can induce a non-vanishing disagreement in the preconditioned update direction, which is a distinct error source from gradient-level client drift. Theorem F.2 (Geometry drift induces preconditioned disagreement) . Consider a round r with a participating client set S_{r} of size S . Each client maintains a (second-order) optimizer state \Theta_{r,k}^{i} and applies the corresponding preconditioner operator P_{\Theta} to a local update vector u_{i} (e.g., stochastic gradient). Assume the operator is Lipschitz in the state: there exists L_{\Theta}>0 such that for any \Theta,\Theta^{\prime} and any v , \|P_{\Theta}(v)-P_{\Theta^{\prime}}(v)\|\leq L_{\Theta}\,\|\Theta-\Theta^{\prime}\|\,\|v\|. (94) Let \bar{\Theta}_{r,K}:=\frac{1}{S}\sum_{i\in S_{r}}\Theta_{r,K}^{i} be the averaged state at the end of local steps, and define the preconditioner drift metric \Delta_{r}^{D}:=\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|\Theta_{r,K}^{i}-\bar{\Theta}_{r,K}\|^{2}. (95) Then the disagreement between using heterogeneous local geometries and the averaged geometry satisfies \mathbb{E}\Big\|\frac{1}{S}\sum_{i\in S_{r}}\big(P_{\Theta_{r,k}^{i}}(u_{i})-P_{\bar{\Theta}_{r,K}}(u_{i})\big)\Big\|^{2}\;\leq\;L_{\Theta}^{2}\Big(\frac{1}{S}\sum_{i\in S_{r}}\mathbb{E}\|u_{i}\|^{2}\Big)\Delta_{r}^{D}. (96) In particular, if \sup_{i,r,k}\mathbb{E}\|u_{i}\|^{2}\leq G^{2} , then \mathbb{E}\Big\|\frac{1}{S}\sum_{i\in S_{r}}\big(P_{\Theta_{r,k}^{i}}(u_{i})-P_{\bar{\Theta}_{r,K}}(u_{i})\big)\Big\|^{2}\;\leq\;L_{\Theta}^{2}G^{2}\,\Delta_{r}^{D}. (97) Corollary F.3 (Drift penalty vanishes under geometry alignment) . If the client geometries are aligned in round r in the sense that \Theta_{r,k}^{i}=\bar{\Theta}_{r,K} for all i\in S_{r} (equivalently, \Delta_{r}^{D}=0 ), then the preconditioned disagreement term in ( 96 ) is zero: \frac{1}{S}\sum_{i\in S_{r}}P_{\Theta_{r,k}^{i}}(u_{i})\;=\;\frac{1}{S}\sum_{i\in S_{r}}P_{\bar{\Theta}_{r,K}}(u_{i}). (98) Consequently, any residual inconsistency across clients is solely due to the vector-level mismatch in \{u_{i}\} (e.g., gradient/client drift from non-IID data), rather than a mismatch in the preconditioning geometry. Remark F.4 (Why geometry drift is distinct from gradient/client drift) . Gradient/client drift concerns the mismatch of the update vectors \{u_{i}\} across clients (e.g., \nabla f_{i}\neq\nabla f ), and control-variate methods aim to reduce this vector-level discrepancy. In contrast, geometry drift concerns a mismatch of the operators \{P_{\Theta_{r,k}^{i}}\} , i.e., clients optimize under different local metrics defined by their second-order states. Theorem shows that even if \{u_{i}\} were identical across clients (no gradient/client drift), heterogeneous geometries ( \Delta_{r}^{D}>0 ) alone induce a non-vanishing disagreement in the preconditioned direction. Corollary further clarifies that synchronizing/aligning \Theta removes this geometry-induced error source, leaving only the conventional gradient/client drift to be handled. Algorithm 4 Local SOAP Algorithm 0: Step of Local SOAP for an m\times n layer. Per layer, we maintain four matrices: L\in\mathbb{R}^{m\times m} , R\in\mathbb{R}^{n\times n} , V,M\in\mathbb{R}^{m\times n} . Hyperparameters: learning rate \eta , betas (\beta_{1},\beta_{2}) , epsilon \epsilon , and preconditioning frequency f , communication rounds T , local updates K , the number of client N . 1: for t=0,\dots,T do 2: for each client i\in\{1,\dots,N\} in parallel do 3: for k=1,\dots,K do 4: Sample batch B_{i}^{t,k} 5: G_{i}^{t,k}\in\mathbb{R}^{m\times n}\leftarrow-\nabla_{W}\phi_{B_{t}}(W_{i}^{t,k}) 6: { End of gradient step, now update L and R and possibly also Q_{L} and Q_{R} } 7: L\leftarrow\beta_{2}L+(1-\beta_{2})(GG^{\top}) 8: R\leftarrow\beta_{2}R+(1-\beta_{2})(G^{\top}G) 9: if t\bmod f=0 then 10: Q_{L}\leftarrow\texttt{Eigenvectors}(L,Q_{L}) 11: Q_{R}\leftarrow\texttt{Eigenvectors}(R,Q_{R}) 12: end if 13: g_{i}^{t,k}\leftarrow Q_{L}^{\top}GQ_{R} 14: 15: UPDATE( g_{i}^{t,k} ) by Adam 16: M_{i}^{t,k}\leftarrow\beta_{1}M_{i}^{t,k}+(1-\beta_{1})g_{i}^{t,k} 17: V_{i}^{t,k}\leftarrow\beta_{2}V_{i}^{t,k}+(1-\beta_{2})(g_{i}^{t,k}\odot g_{i}^{t,k}) 18: N_{i}^{t,k}\leftarrow M_{i}^{t,k}/(\sqrt{V_{i}^{t,k}}+\epsilon) 19: 20: { Now that we have preconditioned by Adam in the rotated space, we go back to the original space } 21: \tilde{N}_{i}^{t,k}\leftarrow Q_{L}N_{i}^{t,k}Q_{R}^{\top} 22: W_{i}^{t,k}\leftarrow W_{i}^{t,k-1}-\eta\tilde{N}_{i}^{t,k} 23: end for 24: Clint i communicate (W_{i}^{t,K}-W_{i}^{t,0}) to Server; 25: end for 26: W^{t+1}\leftarrow W^{t}+\frac{1}{N}\sum_{i}(W_{i}^{t,K}-W_{i}^{t,0}) 27: end for 28: 29: Eigenvectors (P,Q) : 30: S\leftarrow PQ 31: Q\leftarrow\mathrm{QR}(S) {Using QR decomposition} 32: Return Q Algorithm 5 FedPAC_SOAP Algorithm 0: Step of FedPAC_SOAP for an m\times n layer. Per layer, we maintain four matrices: L\in\mathbb{R}^{m\times m} , R\in\mathbb{R}^{n\times n} , V,M\in\mathbb{R}^{m\times n} . Hyperparameters: learning rate \eta , betas (\beta_{1},\beta_{2}) , epsilon \epsilon , and preconditioning frequency f , communication rounds R , local updates K , the number of client N . 1: for r=0,\dots,R do 2: for each client i\in\{1,\dots,N\} in parallel do 3: for k=1,\dots,K do 4: Sample batch B_{i}^{r,k} 5: G_{i}^{r,k}\in\mathbb{R}^{m\times n}\leftarrow-\nabla F_{i}(x_{i}^{r,k};\xi_{i}^{r,k}) 6: { End of gradient step, now update L and R and possibly also Q_{L} and Q_{R} } 7: L\leftarrow\beta_{2}L+(1-\beta_{2})(GG^{\top}) 8: R\leftarrow\beta_{2}R+(1-\beta_{2})(G^{\top}G) 9: if t\bmod f=0 then 10: Q_{L}\leftarrow\texttt{Eigenvectors}(L,Q_{L}) 11: Q_{R}\leftarrow\texttt{Eigenvectors}(R,Q_{R}) 12: end if 13: g_{i}^{t,k}\leftarrow Q_{L}^{\top}GQ_{R} 14: 15: UPDATE( g_{i}^{r,k} ) by Adam 16: M_{i}^{r,k}\leftarrow\beta_{1}M_{i}^{r,k}+(1-\beta_{1})g_{i}^{r,k} 17: V_{i}^{r,k}\leftarrow\beta_{2}V_{i}^{r,k}+(1-\beta_{2})(g_{i}^{r,k}\odot g_{i}^{r,k}) 18: N_{i}^{r,k}\leftarrow M_{i}^{r,k}/(\sqrt{V_{i}^{r,k}}+\epsilon) 19: 20: { Now that we have preconditioned by Adam in the rotated space, we go back to the original space } 21: \tilde{N}_{i}^{r,k}\leftarrow Q_{L}N_{i}^{r,k}Q_{R}^{\top} 22: \boldsymbol{x}^{r,k+1}_{i}\!=\!\boldsymbol{x}^{r,k}_{i}\!\!-\!\eta[(1\!-\!\beta)\tilde{N}_{i}^{r,k}\!+\!\beta\boldsymbol{\Delta}_{G}^{r}] ; 23: end for 24: Client i communicate (x_{i}^{r,K}-x_{i}^{r,0}) to Server; 25: end for 26: \boldsymbol{\Delta}_{G}^{r+1}=-\frac{1}{SK\eta}\sum_{i=1}^{S}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 27: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}+\frac{1}{S}\sum_{i=1}^{S}(\boldsymbol{x}^{r,K}_{i}-\boldsymbol{x}^{r,0}_{i}) ; 28: end for 29: 30: Eigenvectors (P,Q) : 31: S\leftarrow PQ 32: Q\leftarrow\mathrm{QR}(S) {Using QR decomposition} 33: Return Q Algorithm 6 Local Muon Algorithm (Federated Setting) 0: Local Muon for an m\times n weight matrix W (e.g., a Linear layer). Per layer, maintain one matrix: M\in\mathbb{R}^{m\times n} (momentum). Hyperparameters: learning rate \eta , momentum \beta , weight decay \lambda (optional), epsilon \epsilon , Newton–Schulz steps s (typically 5 ), dimension scaling \gamma(m,n) , communication rounds T , local steps K , number of clients N . 1: for t=0,\dots,T do 2: for each client i\in\{1,\dots,N\} in parallel do 3: Initialize local model: W_{i}^{t,0}\leftarrow W^{t} 4: for k=1,\dots,K do 5: Sample batch B_{i}^{t,k} 6: G_{i}^{t,k}\in\mathbb{R}^{m\times n}\leftarrow\nabla_{W}\phi_{B_{i}^{t,k}}(W_{i}^{t,k-1}) 7: M_{i}^{t,k}\leftarrow\beta_{1}\,M_{i}^{t,k-1}+(1-\beta_{1})\,G_{i}^{t,k} 8: { Orthogonalize (typically on momentum) via Newton–Schulz } 9: U_{i}^{t,k}\leftarrow\texttt{NewtonSchulz}(M_{i}^{t,k},s,\epsilon) { U is approximately the orthogonal factor of M } 10: { Apply dimension scaling (one common theoretical form: \gamma(m,n)=\sqrt{m/n} ) } 11: \Delta W_{i}^{t,k}\leftarrow\gamma(m,n)\,U_{i}^{t,k} {e.g. \gamma=\sqrt{\texttt{fan-out}/\texttt{fan-in}} } 12: { Update } 13: W_{i}^{t,k}\leftarrow W_{i}^{t,k-1}-\eta\left(\Delta W_{i}^{t,k}+\lambda W_{i}^{t,k-1}\right) {optional weight decay} 14: end for 15: Client i sends (W_{i}^{t,K}-W_{i}^{t,0}) to Server 16: end for 17: W^{t+1}\leftarrow W^{t}+\frac{1}{N}\sum_{i=1}^{N}(W_{i}^{t,K}-W_{i}^{t,0}) 18: end for 19: 20: NewtonSchulz (G,s,\epsilon) : 21: X\leftarrow G/(\|G\|_{F}+\epsilon) 22: if m>n then X\leftarrow X^{\top} end if 23: for j=1,\dots,s do 24: A\leftarrow XX^{\top} 25: B\leftarrow bA+cA^{2} 26: X\leftarrow aX+BX 27: end for 28: if m>n then X\leftarrow X^{\top} end if 29: return X {common coefficients: a{=}3.4445,\,b{=}{-}4.7750,\,c{=}2.0315 } Algorithm 7 FedPAC_Muon Algorithm 0: Step of FedPAC_Muon for an m\times n layer. Per layer, maintain one matrix: M\in\mathbb{R}^{m\times n} (momentum). Hyperparameters: learning rate \eta , momentum \beta , epsilon \epsilon , Newton–Schulz steps s , (optional) weight decay \lambda , (optional) dimension scaling \gamma(m,n) , communication rounds R , local updates K , number of clients N , FedPAC mixing coefficient \beta\in[0,1] . Server maintains global direction \boldsymbol{\Delta}_{G}^{r} . 1: for r=0,\dots,R do 2: for each client i\in\{1,\dots,N\} in parallel do 3: for k=0,\dots,K-1 do 4: Sample batch B_{i}^{r,k} 5: G_{i}^{r,k}\in\mathbb{R}^{m\times n}\leftarrow-\nabla F_{i}(\boldsymbol{x}_{i}^{r,k};\xi_{i}^{r,k}) 6: M_{i}^{r,k+1}\leftarrow\beta_{1}\,M_{i}^{r,k}+(1-\beta_{1})\,G_{i}^{r,k} 7: { Muon: orthogonalize the (momentum) update direction } 8: U_{i}^{r,k}\leftarrow\texttt{NewtonSchulz}(M_{i}^{r,k+1},s,\epsilon) 9: \tilde{U}_{i}^{r,k}\leftarrow\gamma(m,n)\,U_{i}^{r,k} {optional scaling, e.g. \gamma=\sqrt{m/n} } 10: { FedPAC local update with global direction mixing } 11: \boldsymbol{x}_{i}^{r,k+1}=\boldsymbol{x}_{i}^{r,k}-\eta\Big[(1-\beta)\tilde{U}_{i}^{r,k}+\beta\boldsymbol{\Delta}_{G}^{r}\Big] {optionally add +\lambda\boldsymbol{x}_{i}^{r,k} inside the bracket} 12: end for 13: Client i communicates (\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}_{i}^{r,0}) to Server; 14: end for 15: \boldsymbol{\Delta}_{G}^{r+1}=-\frac{1}{NK\eta}\sum_{i=1}^{N}\big(\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}_{i}^{r,0}\big) ; 16: \boldsymbol{x}^{r+1}=\boldsymbol{x}^{r}+\frac{1}{N}\sum_{i=1}^{N}\big(\boldsymbol{x}_{i}^{r,K}-\boldsymbol{x}_{i}^{r,0}\big) ; 17: end for 18: 19: NewtonSchulz (G,s,\epsilon) : 20: X\leftarrow G/(\|G\|_{F}+\epsilon) 21: if m>n then X\leftarrow X^{\top} end if 22: for j=1,\dots,s do 23: A\leftarrow XX^{\top} 24: X\leftarrow\frac{1}{2}X(3I-A) {classic Newton–Schulz for (XX^{\top})^{-1/2} } 25: end for 26: if m>n then X\leftarrow X^{\top} end if 27: return X Algorithm 8 Local Sophia Algorithm (Federated Setting) 0: Local Sophia for an m\times n layer W . Per layer, maintain two matrices: M,H\in\mathbb{R}^{m\times n} (momentum and Hessian-diagonal EMA). Hyperparameters: learning rate \eta , betas (\beta_{1},\beta_{2}) , epsilon \epsilon , clipping \rho , Hessian update frequency f_{h} , communication rounds T , local steps K , number of clients N , weight decay \lambda (optional). 1: for t=0,\dots,T do 2: for each client i\in\{1,\dots,N\} in parallel do 3: Initialize local model: W_{i}^{t,0}\leftarrow W^{t} 4: for k=1,\dots,K do 5: Sample batch B_{i}^{t,k} 6: G_{i}^{t,k}\leftarrow\nabla_{W}\phi_{B_{i}^{t,k}}(W_{i}^{t,k-1}) 7: M_{i}^{t,k}\leftarrow\beta_{1}M_{i}^{t,k-1}+(1-\beta_{1})G_{i}^{t,k} 8: { Update H (diagonal Hessian estimate) occasionally } 9: if ((tK+k)\bmod f_{h})=0 then 10: Sample Rademacher noise U_{i}^{t,k}\in\{\pm 1\}^{m\times n} 11: HV_{i}^{t,k}\leftarrow\nabla_{W}^{2}\phi_{B_{i}^{t,k}}(W_{i}^{t,k-1})\,U_{i}^{t,k} {HVP via auto-diff (Pearlmutter trick)} 12: \widehat{H}_{i}^{t,k}\leftarrow U_{i}^{t,k}\odot HV_{i}^{t,k} {Unbiased diag(H) estimator} 13: H_{i}^{t,k}\leftarrow\beta_{2}H_{i}^{t,k-1}+(1-\beta_{2})\max(\widehat{H}_{i}^{t,k},0) 14: else 15: H_{i}^{t,k}\leftarrow H_{i}^{t,k-1} 16: end if 17: D_{i}^{t,k}\leftarrow M_{i}^{t,k}\oslash\max(H_{i}^{t,k},\epsilon) {element-wise divide} 18: D_{i}^{t,k}\leftarrow\texttt{clip}(D_{i}^{t,k},-\rho,\rho) 19: W_{i}^{t,k}\leftarrow W_{i}^{t,k-1}-\eta\left(D_{i}^{t,k}+\lambda W_{i}^{t,k-1}\right) {optional weight decay} 20: end for 21: Client i sends (W_{i}^{t,K}-W_{i}^{t,0}) to Server 22: end for 23: W^{t+1}\leftarrow W^{t}+\frac{1}{N}\sum_{i=1}^{N}(W_{i}^{t,K}-W_{i}^{t,0}) 24: end for 25: 26: clip (X,a,b) : element-wise \min(\max(X,a),b) . Algorithm 9 PAC_Sophia Algorithm (Federated Setting) 0: PAC_Sophia for an m\times n layer W . Per layer, maintain two matrices: M,H\in\mathbb{R}^{m\times n} (momentum and Hessian-diagonal EMA). Hyperparameters: learning rate \eta , betas (\beta_{1},\beta_{2}) , epsilon \epsilon , clipping \rho , Hessian update frequency f_{h} , communication rounds R , local steps K , number of clients N , PAC mixing coefficient \beta\in[0,1] , weight decay \lambda (optional). Server maintains global direction \boldsymbol{\Delta}_{G}^{r} . 1: for r=0,\dots,R do 2: for each client i\in\{1,\dots,N\} in parallel do 3: Initialize local model: W_{i}^{r,0}\leftarrow W^{r} 4: for k=1,\dots,K do 5: Sample batch B_{i}^{r,k} 6: G_{i}^{r,k}\leftarrow\nabla_{W}\phi_{B_{i}^{r,k}}(W_{i}^{r,k-1}) 7: M_{i}^{r,k}\leftarrow\beta_{1}M_{i}^{r,k-1}+(1-\beta_{1})G_{i}^{r,k} 8: { Update H (diagonal Hessian estimate) occasionally } 9: if ((rK+k)\bmod f_{h})=0 then 10: Sample Rademacher noise U_{i}^{r,k}\in\{\pm 1\}^{m\times n} 11: HV_{i}^{r,k}\leftarrow\nabla_{W}^{2}\phi_{B_{i}^{r,k}}(W_{i}^{r,k-1})\,U_{i}^{r,k} {HVP via auto-diff (Pearlmutter trick)} 12: \widehat{H}_{i}^{r,k}\leftarrow U_{i}^{r,k}\odot HV_{i}^{r,k} {Unbiased diag(H) estimator} 13: H_{i}^{r,k}\leftarrow\beta_{2}H_{i}^{r,k-1}+(1-\beta_{2})\max(\widehat{H}_{i}^{r,k},0) 14: else 15: H_{i}^{r,k}\leftarrow H_{i}^{r,k-1} 16: end if 17: D_{i}^{r,k}\leftarrow M_{i}^{r,k}\oslash\max(H_{i}^{r,k},\epsilon) 18: D_{i}^{r,k}\leftarrow\texttt{clip}(D_{i}^{r,k},-\rho,\rho) {Sophia direction} 19: { PAC mixing with global direction } 20: W_{i}^{r,k}\leftarrow W_{i}^{r,k-1}-\eta\Big[(1-\beta)D_{i}^{r,k}+\beta\boldsymbol{\Delta}_{G}^{r}+\lambda W_{i}^{r,k-1}\Big] { \lambda optional} 21: end for 22: Client i communicates (W_{i}^{r,K}-W_{i}^{r,0}) to Server 23: end for 24: \boldsymbol{\Delta}_{G}^{r+1}=-\frac{1}{NK\eta}\sum_{i=1}^{N}\big(W_{i}^{r,K}-W_{i}^{r,0}\big) ; 25: W^{r+1}=W^{r}+\frac{1}{N}\sum_{i=1}^{N}\big(W_{i}^{r,K}-W_{i}^{r,0}\big) ; 26: end for 27: 28: clip (X,a,b) : element-wise \min(\max(X,a),b) .