Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment Hongyi Peng, Han Yu, Xiaoxiao Li, Qiang Yang Abstract Low-Rank Adaptation (LoRA) is widely used for federated fine-tuning. Yet under non-IID settings, it can substantially underperform full-parameter fine-tuning. Through with-high-probability robustness analysis, we uncover that this gap can be attributed to two coupled mismatches: (i) update-space mismatch , where clients optimize in a low-rank subspace but aggregation occurs in the full space; and (ii) optimizer-state mismatch , where unsynchronized adaptive states amplify drift across rounds. We propose FedGaLore , which combines client-side GaLore-style gradient-subspace optimization with server-side drift-robust synchronization of projected second-moment states via spectral shared-signal extraction, to address this challenge. Across NLU, vision, and NLG benchmarks, FedGaLore improves robustness and accuracy over state-of-the-art federated LoRA baselines in non-IID settings. 1 Introduction Foundation models (FMs) have reshaped machine learning via the pretrain–finetune paradigm ( 5 ; 6 ; 54 ; 26 ) . However, their scale ( 23 ) makes full fine-tuning (FFT) prohibitively expensive. This has motivated parameter-efficient fine-tuning (PEFT), where Low-Rank Adaptation (LoRA) ( 22 ) has become the de facto standard. LoRA freezes pretrained weights and learns low-rank adapter factors {\bm{A}} and {\bm{B}} , thereby greatly reducing trainable parameters (and optimizer-state memory) while retaining strong centralized performance. These benefits have spurred federated LoRA variants. However, federated learning (FL) is sensitive to statistical heterogeneity: non-IID clients might produce biased and divergent local updates. As a result, while LoRA can match the performance of FFT in centralized training ( 22 ) , it often degrades substantially under non-IID federated settings ( 3 ; 63 ) . Figure 1 : Left: Under increasing data heterogeneity (smaller Dirichlet \alpha ), representative federated LoRA baselines degrade sharply, while FedGaLore remains stable and approaches full fine-tuning performance. Right: With local adaptive optimizers, client training loss can decrease while global validation performance stagnates or degrades, indicating optimizer-state mismatch. Figure 1 shows that federated LoRA fine-tuning underperforms full fine-tuning under non-IID FL settings. The gap widens as data heterogeneity increases. Prior work mitigates this by further restricting the update space (e.g., FFA-LoRA shares a factor across clients), which can improve robustness under severe data heterogeneity, with reduced IID performance ( 52 ) . We also observe a local–global mismatch with adaptive optimizers (e.g., Adam/AdamW): local training loss decreases while global validation performance improves little or even degrades. These observations point to two coupled mechanisms: (i) misalignment of client updates in the parameter space, and (ii) drift of adaptive optimizer states across clients and rounds. To formalize this and complement standard expectation-based analysis, we develop a with-high-probability (W.H.P.) robustness framework that controls the probability of rare but catastrophic deviations under heterogeneity. We view federated fine-tuning as a perturbed implementation of a centralized optimizer, and decompose each round into client training ( \mathcal{T} ), server aggregation ( \mathcal{A} ) and state synchronization ( \mathcal{S} ). This lens isolates two failure modes: (1) Update-space mismatch: LoRA constrains each client update to a low-rank subspace, while \mathcal{A} aggregates in the full parameter space. (2) Optimizer-state mismatch: inconsistencies in adaptive states (arising from imperfect \mathcal{S} and amplified by \mathcal{T} ) propagate through local dynamics and amplify client drift. Together, these effects explain the brittleness of federated LoRA under data heterogeneity. To address these limitations, we propose FedGaLore . It couples gradient-subspace client optimization with drift-robust state synchronization . On the client side, we replace LoRA’s fixed parameter-subspace updates with GaLore-style gradient-subspace optimization: each client computes gradients on the adapted modules and updates in an adaptively estimated rank- r subspace, preserving LoRA-like parameter efficiency while improving aggregation robustness. Clients use a GaLore-based AdamW variant to maintain local adaptivity. On the server side, we mitigate optimizer-state mismatch with minimal communication by synchronizing only the projected second-moment state that defines the preconditioner. Specifically, clients upload low-rank projected second moments, and the server applies an AJIVE filter ( 14 ) to extract the shared component across clients and broadcasts it as the next-round initialization, stabilizing training under non-IID data. Extensive experimental evaluation has been conducted on the NLU, vision and NLG benchmarks. The results show that FedGaLore significantly improves robustness to data heterogeneity and stability over state-of-the-art federated LoRA baselines. 2 Related Works LoRA ( 22 ) has inspired many extensions, including rank-adaptive/structured adapters ( 71 ; 27 ; 34 ) and improved initialization ( 31 ; 41 ) . More recently, GaLore ( 72 ) shifts from parameter-space adapters to gradient-subspace optimization : it projects gradients into an evolving low-rank subspace, preserving access to full-gradient information while keeping updates low-rank. This has motivated follow-up work on theory and variants ( 20 ; 42 ; 50 ) , and comparative analyses suggesting more favorable optimization behavior than fixed parameter constraints in LoRA ( 19 ; 35 ) . FL must cope with client drift induced by statistical heterogeneity. FedAvg ( 40 ) and drift-control methods such as SCAFFOLD ( 24 ) and FedDyn ( 1 ) primarily analyze local SGD, while FedOpt introduces server-side adaptivity ( 47 ) . More recent work studies local adaptive optimizers ( 29 ; 53 ) , but synchronizing their internal states remains nontrivial ( 12 ) . On the theory side, convergence analyses increasingly cover broader heterogeneity ( 8 ) , yet most results are in expectation and can obscure rare but destabilizing events. We adopt a with-high-probability (W.H.P.) lens ( 10 ) to explicitly control such outliers, which is particularly relevant for robust federated fine-tuning. Federated LoRA is not a drop-in replacement for FFT-based FL: heterogeneity remedies developed for full fine-tuning do not necessarily transfer because LoRA changes the update space and effective local capacity. Empirically, combining LoRA with standard drift-control can be inconsistent ( 67 ) . Moreover, many federated LoRA methods adopt local adaptive optimizers (Adam/AdamW) for faster client progress, but rarely account for optimizer-state drift across clients and rounds. Table 1 highlights that prior work varies in optimizer choice and whether aggregation is performed on low-rank factors or lifted full-space updates, yet few explicitly synchronize adaptive states. Table 1 : Representative federated LoRA methods. Local optimizer, aggregation, and whether optimizer states are synchronized. Method Opt. Agg. Sync FedIT ( 70 ) Adam factors No FFA-LoRA ( 52 ) SGD factors No LoRA-Fair ( 4 ) SGD factors No FLoRA ( 60 ) AdamW lift \Delta W No FR-LoRA ( 64 ) AdamW lift \Delta W No 3 Preliminaries Notation. For a\in\mathbb{R}^{d} , \|a\| denotes the Euclidean norm and \|a\|_{\infty}\coloneqq\max_{i}|a_{i}| the infinity norm. For vectors, \odot and / denote element-wise multiplication and division; a^{2} , \sqrt{a} , and |a| are applied coordinate-wise. Matrices are denoted by \bm{A}\in\mathbb{R}^{d_{1}\times d_{2}} ; \|\bm{A}\| is the spectral norm and \|\bm{A}\|_{F} the Frobenius norm. Problem Setup We consider an FL system with M clients, where client i has a local dataset \mathcal{D}_{i}=\{(x_{n},y_{n})\}_{n=1}^{N_{i}} and objective F_{i}(\theta) (population or empirical risk). Client data may be non-IID: each client draws samples from a potentially different distribution \mathbb{P}_{i} , inducing biased and divergent local updates under heterogeneous training. Let \theta\in\Theta\subseteq\mathbb{R}^{d} denote model parameters and define the global objective f(\theta)\coloneqq\sum_{i=1}^{M}p_{i}F_{i}(\theta) , where p_{i}\geq 0 , \sum_{i=1}^{M}p_{i}=1 , and p_{i} is typically proportional to data size or client sampling probability. 1 1 1 We assume full participation unless stated otherwise. Let g^{i,k}_{t} denote the stochastic gradient on client i at round k and local step t (an unbiased estimator of \nabla F_{i}(\theta^{i,k}_{t}) ). We decompose: g^{i,k}_{t}\;=\;\nabla f(\theta^{i,k}_{t})\;+\;c^{i,k}_{t}\;+\;\xi^{i,k}_{t}. (1) c^{i,k}_{t}\coloneqq\nabla F_{i}(\theta^{i,k}_{t})-\nabla f(\theta^{i,k}_{t}) captures client drift. \xi^{i,k}_{t}\coloneqq g^{i,k}_{t}-\nabla F_{i}(\theta^{i,k}_{t}) is mini-batch noise. For any fixed \theta , \sum_{i=1}^{M}p_{i}\big(\nabla F_{i}(\theta)-\nabla f(\theta)\big)=0 by definition. Under local training c^{i,k}_{t} is evaluated along different client trajectories. Definition 3.1 (Local Training Operator) . The local training operator \mathcal{T}_{i}(\bar{\theta}_{k},S^{i,k}_{0};T) runs T local steps on client i starting from global model \bar{\theta}_{k} and initial optimizer state S^{i,k}_{0} (e.g., SGD, Adam ( 25 ) , AdamW ( 38 ) ). It outputs (\theta^{i,k}_{T},S^{i,k}_{T})\;=\;\mathcal{T}_{i}(\bar{\theta}_{k},S^{i,k}_{0};T). (2) Definition 3.2 (Server Aggregation Operator) . Given participating clients \mathcal{P}_{k}\subseteq[M] and their terminal models \{\theta^{i,k}_{T}\}_{i\in\mathcal{P}_{k}} , the aggregation operator \mathcal{A} produces \bar{\theta}_{k+1}\;=\;\mathcal{A}\big(\{\theta^{i,k}_{T}\}_{i\in\mathcal{P}_{k}}\big), (3) with a canonical choice being FedAvg: \bar{\theta}_{k+1}=\sum_{i\in\mathcal{P}_{k}}\tilde{p}_{i}\,\theta^{i,k}_{T} , where \tilde{p}_{i}\coloneqq\frac{p_{i}}{\sum_{j\in\mathcal{P}_{k}}p_{j}} . Definition 3.3 (State Synchronization Protocol) . The synchronization operator \mathcal{S} specifies how optimizer states are shared or aggregated across rounds. Given terminal client states \{S^{i,k}_{T}\}_{i\in\mathcal{P}_{k}} (and optionally server state \bar{S}_{k} ): \displaystyle\bar{S}_{k+1}\; \displaystyle=\;\mathcal{S}\big(\{S^{i,k}_{T}\}_{i\in\mathcal{P}_{k}},\bar{S}_{k}\big), (4) \displaystyle S^{i,k+1}_{0} \displaystyle\leftarrow\mathrm{InitState}(\bar{S}_{k+1}). (5) Common choices: 1) none (clients reinitialize each round), and 2) server-only (the server maintains adaptive states). This modular view covers many FL algorithms. FedAvg ( 40 ) uses SGD for \mathcal{T} , FedAvg for \mathcal{A} , and none for \mathcal{S} . SCAFFOLD ( 24 ) introduces control variates in \mathcal{T} that must be synchronized by \mathcal{S} . FedOpt/FedAdam ( 47 ) typically keeps \mathcal{T} stateless while implementing adaptivity at the server through \mathcal{A} together with server-maintained states in \mathcal{S} . Assumptions We analyze federated fine-tuning over a region Q\subseteq\Theta that contains the iterates (i.e., \theta^{i,k}_{t}\in Q for all i,k,t ). Unless stated otherwise, all assumptions below hold on Q . This restricted-region viewpoint is standard for modern over-parameterized networks: although the global landscape is non-convex, the optimizer often remains in a well-behaved region where a local PL condition holds (often termed PL ∗ ) ( 32 ) . Assumption 3.4 (Lower-boundedness) . The objective is lower bounded on Q : f^{*}\coloneqq\inf_{\theta\in Q}f(\theta)>-\infty . Assumption 3.5 (Smoothness) . Each client objective F_{i} is L -smooth on Q : \|\nabla F_{i}(x)-\nabla F_{i}(y)\|\leq L\|x-y\|,\quad\forall x,y\in Q,\ \forall i\in[M]. (6) In particular, f=\sum_{i}p_{i}F_{i} is also L -smooth on Q . Assumption 3.6 (Polyak–Łojasiewicz (PL)) . There exists \mu>0 such that for all \theta\in Q , \|\nabla f(\theta)\|^{2}\geq 2\mu\big(f(\theta)-f^{*}\big). (7) Assumption 3.7 (Bounded heterogeneity) . There exist constants B\geq 1 and H\geq 0 such that for all \theta\in Q , \sum_{i=1}^{M}p_{i}\|\nabla F_{i}(\theta)\|^{2}\leq H^{2}+B^{2}\|\nabla f(\theta)\|^{2}. (8) Assumption 3.8 (Bounded local gradients) . There exists G\geq 0 such that for all i\in[M] and \theta\in Q , \|\nabla F_{i}(\theta)\|\leq G . Assumption 3.7 is standard in FL for controlling non-IID drift ( 24 ; 58 ) . Assumption 3.8 is a technical condition for bounded increments in high-probability analyses and is satisfied when using norm-wise gradient clipping with threshold G ( 17 ; 47 ; 62 ; 59 ) . Assumption 3.9 (Sub-Gaussian mini-batch noise) . For all (i,k,t) , the noise \xi_{t}^{i,k} is conditionally mean-zero and \sigma -sub-Gaussian given \mathcal{F}_{k,t-1} (Definition B.1 ). This assumption is standard for W.H.P. bounds via martingale concentration ( 48 ) . It is also common in stochastic-dynamics views of optimization ( 61 ; 46 ) . Our results can be extended to weaker moment assumptions following, e.g., 7 . 4 A W.H.P. Lens for Federated Learning Why W.H.P. robustness? Expectation-based analyses describe average behavior but can obscure instability under severe heterogeneity. In federated fine-tuning, client drift and mini-batch noise can interact to produce rare but catastrophic trajectories that leave a stable basin and derail aggregation. With-high-probability (W.H.P.) guarantees address this by controlling the entire optimization path with confidence 1-\delta over the stochasticity. For instance, if a method converges for 90\% of realizations but diverges for the remaining 10\% , an in-expectation guarantee may still look favorable, whereas a W.H.P. statement cannot hold at confidence levels above 0.9 . Q (Stable Region) \theta^{\star} (Reference) \bar{\theta}^{k} \mathcal{A} Sync \bar{\theta}^{k+1} Escape \mathcal{E}_{\mathrm{traj}}^{c} \mathcal{A} \bar{\theta}^{k+2} \mathcal{A} Unstable Agg. Output \mathcal{E}_{\mathrm{agg}}^{c} Figure 2 : Trajectory robustness and failure modes. The stable region Q (blue tube) surrounds a centralized reference solution \theta^{\star} (dashed). Left (round k ): Stable operation; all client trajectories remain in Q . Middle (round k{+}1 ): Local escape ( \mathcal{E}_{\mathrm{traj}}^{c} , red), where client drift causes a trajectory to exit Q . Right (round k{+}2 ): Aggregation instability ( \mathcal{E}_{\mathrm{agg}}^{c} ), where the aggregated model leaves Q even though all client endpoints lie in Q . Inspired by recent W.H.P. analyses of local adaptive optimization ( 7 ) , we study FL through a trajectory-robustness lens. We compare federated iterates \{\bar{\theta}^{k}\}_{k\geq 0} to a centralized reference trajectory \{\theta_{t}^{\star}\} within a stable region Q where f is well behaved (Assumptions 3.5 – 3.6 ). For round k , let \Theta^{i,k} denote client i ’s local trajectory. As illustrated in Fig. 2 , we decompose stability into two coupled success events: • Local containment \mathcal{E}_{\mathrm{traj}}(K) : all client trajectories stay in Q for rounds k\leq K . • Aggregation stability \mathcal{E}_{\mathrm{agg}}(K) : the aggregated iterate \bar{\theta}^{k+1} stays in Q for rounds k\leq K . Controlling the failure probabilities of \mathcal{E}_{\mathrm{traj}}(K) and \mathcal{E}_{\mathrm{agg}}(K) implies, via a union bound and induction, that the global trajectory remains in Q with probability at least 1-\delta , enabling standard contraction arguments throughout training. This modular roadmap isolates failure modes: \mathcal{E}_{\mathrm{traj}} reflects drift and optimizer-state mismatch, while \mathcal{E}_{\mathrm{agg}} captures aggregation instability. 4.1 Update-space Mismatch We analyze aggregation stability, which governs \mathcal{E}_{\mathrm{agg}}(K) : even if all clients remain in a stable region, aggregation might push the global iterate outside it. Given round k and assume \theta^{i,k}_{T}\in Q for all participating clients, the question is whether the server update \bar{\theta}^{k+1}=\mathcal{A}(\{\theta^{i,k}_{T}\}) still lies in Q . Under convexity this is immediate: if Q is convex and \mathcal{A} is a convex combination, aggregation cannot leave Q . Lemma 4.1 (Aggregation preserves convex regions) . Let Q be convex. If \theta_{i}\in Q for all i\in[M] and \bar{\theta}=\sum_{i=1}^{M}\tilde{p}_{i}\theta_{i} with \tilde{p}_{i}\geq 0 and \sum_{i}\tilde{p}_{i}=1 , then \bar{\theta}\in Q . Lemma 4.1 explains why aggregation is rarely a bottleneck in convex analyses, but this closure property typically fails for deep networks where stable regions (e.g., PL basins) are local and non-convex ( 32 ) . Empirically, mode-connectivity results show that independently trained solutions can often be connected by low-loss paths, sometimes making linear interpolation near-optimal ( 16 ; 13 ) . This motivates weight averaging and related model-merging methods ( 2 ; 65 ) , and is supported by recent theory under additional conditions ( 15 ; 28 ) . Together, these observations help explain why full-parameter FedAvg is often stable: weighted averages of nearby solutions frequently remain within (or close to) a well-behaved region. Federated LoRA differs from FFT. For a linear module, LoRA parameterizes \bm{W}=\bm{W}^{(0)}+\bm{B}\bm{A} with rank r\ll\min\{d_{\mathrm{out}},d_{\mathrm{in}}\} , so each client update \Delta\bm{W}=\bm{B}\bm{A} lies in a nonconvex rank- \leq r set. Early federated LoRA methods aggregate factors : FedIT ( 70 ) uses \Delta\bar{\bm{W}}^{k}=(\sum_{i}\tilde{p}_{i}\,\bm{B}^{i,k}_{T})(\sum_{i}\tilde{p}_{i}\,\bm{A}^{i,k}_{T}) , and FFA-LoRA ( 52 ) fixes one factor so \Delta\bar{\bm{W}}^{k}=(\sum_{i}\tilde{p}_{i}\,\bm{B}^{i,k}_{T})\bm{A}^{(0)} . These enforce a rank- \leq r aggregate ( low-rank-space aggregation ). In contrast, full-space aggregation lifts each client adapter to a weight delta, and averages in the ambient space ( 64 ; 60 ) : \Delta\bar{\bm{W}}^{k}=\sum_{i\in\mathcal{P}_{k}}\tilde{p}_{i}\,(\bm{B}^{i,k}_{T}\bm{A}^{i,k}_{T}), so \mathrm{rank}(\Delta\bar{\bm{W}}^{k}) can grow up to |\mathcal{P}_{k}|r even when each client update is rank- \leq r , injecting out-of-subspace components that can destabilize aggregation. To formalize the geometric tension, consider the rank- r manifold (away from rank-deficient points) \mathcal{M}_{r}\coloneqq\{\Delta\bm{W}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}:\ \mathrm{rank}(\Delta\bm{W})=r\}, with ambient dimension D_{\mathrm{amb}}=d_{\mathrm{out}}d_{\mathrm{in}} , manifold dimension d_{\mathcal{M}}=r(d_{\mathrm{out}}+d_{\mathrm{in}}-r) , and codimension \mathrm{codim}(\mathcal{M}_{r})=(d_{\mathrm{out}}-r)(d_{\mathrm{in}}-r) . To make volumes finite, we restrict to a bounded Frobenius ball \mathbb{B}_{F}(\rho) and define the tube \mathcal{M}_{r}^{(R)}\coloneqq\{\Delta\bm{W}\in\mathbb{B}_{F}(\rho):\ \mathrm{dist}_{F}(\Delta\bm{W},\mathcal{M}_{r})\leq R\}. Theorem 4.2 (Weyl’s tube formula (informal) ( 18 ) ) . Let \mathcal{M} be a compact d_{\mathcal{M}} -dimensional C^{2} submanifold of \mathbb{R}^{D_{\mathrm{amb}}} . For sufficiently small R , \displaystyle\mathrm{Vol}(\mathcal{M}^{(R)})= \displaystyle\mathrm{Vol}(\mathcal{M})\,\mathrm{Vol}\!\big(B^{D_{\mathrm{amb}}-d_{\mathcal{M}}}(R)\big) (9) \displaystyle+ \displaystyle O\!\big(R^{D_{\mathrm{amb}}-d_{\mathcal{M}}+1}\big), so \mathrm{Vol}(\mathcal{M}^{(R)}) scales as R^{D_{\mathrm{amb}}-d_{\mathcal{M}}} to leading order. Theorem 4.2 highlights an unforgiving geometry: when \mathrm{codim}(\mathcal{M}_{r}) is large, an R -tube around rank- r updates occupies a vanishingly small fraction of the ambient space, so small off-manifold components can push \Delta\bar{\bm{W}}^{k} outside a “stable tube” Q . A complementary view follows from Eckart–Young. Let \mathcal{M}_{\leq r}\coloneqq\{\Delta\bm{W}:\mathrm{rank}(\Delta\bm{W})\leq r\} : \mathrm{dist}_{F}(\Delta\bar{\bm{W}}^{k},\mathcal{M}_{\leq r})=\Big(\sum_{j>r}\sigma_{j}(\Delta\bar{\bm{W}}^{k})^{2}\Big)^{1/2}, (10) where \{\sigma_{j}(\cdot)\} are singular values in nonincreasing order; thus misaligned client updates induce nontrivial tail singular values beyond rank r . We refer to this failure mode as update-space mismatch , which helps explain why more restrictive schemes (e.g., fixing one adaptation factor in FFA-LoRA ( 52 ) ) can trade expressivity for stability as non-IID severity increases. 4.2 Optimizer-state Mismatch While update-space mismatch governs aggregation stability (event \mathcal{E}_{\mathrm{agg}} ), local containment (event \mathcal{E}_{\mathrm{traj}} ) is governed by the stochastic dynamics of client-side optimization. Adaptive optimizers (e.g., Adam/AdamW ( 25 ; 22 ) ) are the de facto standard for fine-tuning large models; LoRA typically uses AdamW. In FL, clients optimize different non-IID objectives, so their adaptive states (e.g., first/second moments) evolve inconsistently across clients and rounds. We term this phenomenon optimizer-state mismatch . Such mismatch can bias local update directions and amplify drift/noise, increasing the probability that a client trajectory exits the stable region (event \mathcal{E}_{\mathrm{traj}}^{c} ). Since most federated LoRA methods do not explicitly synchronize adaptive states, characterizing state mismatch propagation through local dynamics is critical. We compare each client to a centralized reference optimizer, which induces a canonical state (m^{\star,k}_{0},v^{\star,k}_{0}) at the start of round k . Practical protocols approximate these states (e.g., pseudo-gradients from deltas ( 47 ; 12 ) or state aggregation ( 51 ; 33 ) ); we model the resulting discrepancy as a bounded initialization error. Definition 4.3 (Biased state initialization) . At the start of round k , client i initializes (m^{i,k}_{0},v^{i,k}_{0}) while the reference uses (m^{\star,k}_{0},v^{\star,k}_{0}) . Assume \|m^{i,k}_{0}-m^{\star,k}_{0}\|\leq B_{m} and \|v^{i,k}_{0}-v^{\star,k}_{0}\|\leq B_{v} for all i,k , where \|\cdot\| denotes \ell_{2} for vectors and Frobenius norm for matrices (set B_{v}=0 for optimizers without v ). Let \varepsilon_{\mathrm{noise}}(\delta) denote the W.H.P. bound from Lemma B.2 such that, with probability at least 1-\delta , \max_{k\leq K-1,\;t\leq T,\;i\in[M]}\|\xi_{t}^{i,k}\|_{2}\leq\varepsilon_{\mathrm{noise}}(\delta). Theorem 4.4 (W.H.P. local-containment radius) . Assume (i) Assumptions 3.5 and 3.8 , (ii) the noise envelope above, and (iii) Definition 4.3 . Let \{\theta_{t}^{\star,k}\}_{t=0}^{T} be the within-round reference trajectory started from \bar{\theta}^{k} under the corresponding centralized update rule. Suppose \eta\leq\frac{1}{2LT} and standard hyperparameters are used (Appendix A ). Then for any \delta\in(0,1) , with probability at least 1-\delta over mini-batch sampling, simultaneously for all rounds k\leq K-1 , all clients i\in[M] , and all local steps t\leq T , \|\theta^{i,k}_{t}-\theta^{\star,k}_{t}\|_{2}\leq\mathcal{R}_{\mathrm{loc}}(\delta), where, up to absolute constants, \mathcal{R}_{\mathrm{loc}}(\delta)\;\lesssim\;\mathcal{R}_{\mathrm{drift}}(\delta)+\mathcal{R}_{\mathrm{state}}, with \mathcal{R}_{\mathrm{drift}}(\delta)=\begin{cases}\eta T\big(G+\varepsilon_{\mathrm{noise}}(\delta)\big),&\mathrm{SGD/Momentum},\\[2.0pt] \frac{\eta T}{\sqrt{\epsilon}}\big(G+\varepsilon_{\mathrm{noise}}(\delta)\big),&\mathrm{AdamW},\end{cases} and \mathcal{R}_{\mathrm{state}}=\begin{cases}0,&\mathrm{SGD},\\[2.0pt] \eta\,\dfrac{B_{m}}{1-\beta_{1}},&\mathrm{Momentum},\\[6.0pt] \eta\!\left(\dfrac{B_{m}}{(1-\beta_{1})\sqrt{\epsilon}}+\dfrac{G\,B_{v}}{(1-\beta_{2})\epsilon^{3/2}}\right),&\mathrm{AdamW}.\end{cases} Here AdamW uses the elementwise preconditioner (v+\epsilon)^{-1/2} with \epsilon>0 . We defer the proof to Appendix B.2 . Theorem 4.4 shows that optimizer-state mismatch directly inflates the local containment radius, increasing the chance of local escape and imposing a stricter geometric requirement on aggregation stability. Notably, the second-moment mismatch term is amplified by the preconditioner: small perturbations in v translate into larger changes in (v+\epsilon)^{-1/2} , which motivates prioritizing second-moment synchronization, consistent with ( 33 ; 51 ) . Moreover, under bounded heterogeneity (Assumption 3.7 ), worst-case drift control can be replaced by an average/RMS drift bound, yielding improved in-expectation convergence guarantees (Corollary B.3 in Appendix). 5 The Proposed FedGaLore Method (a) GaLore (Synthetic) (b) LoRA (Synthetic) (c) Linear Connectivity (ViT) Figure 3 : Geometric robustness and aggregation stability. (a–b) Synthetic landscape: GaLore-style updates reach flat basins more often than fixed-subspace LoRA (60% vs. 20%; Appendix C ). (c) Non-IID ViT-MNIST: linear interpolation between client models shows low loss barriers for FFT and GaLore-style updates, while LoRA exhibits a higher barrier, indicating reduced aggregation robustness. Our analysis identifies two coupled failure modes in federated PEFT under heterogeneity: (1) Update-space mismatch: FFT is often aggregation-robust but expensive, whereas LoRA constrains updates to a fixed low-rank manifold, making aggregation brittle under non-IID. (2) Optimizer-state mismatch: adaptive optimizers speed up local training, but unsynchronized second-moment states amplify drift via preconditioning, degrading local containment and aggregation stability. FedGaLore addresses both by combining (i) client-side gradient-subspace optimization with low-rank states and (ii) server-side drift-robust synchronization of second-moment matrices via spectral shared-signal extraction. Algorithm 1 summarizes one round. Client-side optimization (GaLore on target modules). FedGaLore applies GaLore-style gradient-subspace optimization to target weight matrices (e.g., attention \{\bm{W}_{Q},\bm{W}_{K},\bm{W}_{V},\bm{W}_{O}\} and MLP projection matrices), each treated as a matrix block \bm{W}\in\mathbb{R}^{n\times n} for exposition. 2 2 2 Rectangular blocks are handled analogously. In AdamW, the second-moment state is elementwise; we write \tilde{\bm{v}} for the elementwise buffer reshaped to match the projected tensor. At local step t , client i maintains a rank- r projector \bm{P}^{i,k}_{t}\in\mathbb{R}^{r\times n} and computes the dense gradient \bm{g}^{i,k}_{t}=\nabla_{\bm{W}}\ell(\theta^{i,k}_{t};\mathcal{B}^{i,k}_{t})\in\mathbb{R}^{n\times n} . Under default GaLore ( proj_type=std ), we use a one-sided right projection \tilde{\bm{g}}^{i,k}_{t}=\bm{g}^{i,k}_{t}(\bm{P}^{i,k}_{t})^{\top}\in\mathbb{R}^{n\times r}, and maintain projected optimizer buffers (\tilde{\bm{m}}^{i,k}_{t},\tilde{\bm{v}}^{i,k}_{t})\in\mathbb{R}^{n\times r} , reprojecting them whenever \bm{P}^{i,k}_{t} is refreshed (every \tau steps). SVD-to-random projectors. Following prior work ( 42 ) , we use data-driven projectors (randomized SVD/RSVD) in early local epochs and switch to a random orthonormal projector in later epochs to reduce overhead. In the random phase, the projector is fully determined by an integer seed . Accordingly, the server broadcasts a per-round seed s_{k} enabling clients to deterministically construct the same projector without transmitting basis matrices. We show in Appendix D that this schedule improves wall-clock efficiency and accelerates local training convergence. What clients send. After T local steps, client i uploads (i) a rank- r factorized model update for the adapted blocks (or equivalently \Delta\theta_{i}=\theta^{i,k}_{T}-\bar{\theta}_{k} in compressed form), and (ii) the projected second-moment buffer \tilde{\bm{v}}^{i,k}_{T}\in\mathbb{R}^{n\times r} used for server-side synchronization. This differs from LoRA in a crucial way: LoRA fixes a low-rank parameterization , whereas FedGaLore adapts a low-rank update subspace estimated from recent gradients. As a result, clients can exploit informative directions outside any fixed low-rank manifold, producing trajectories that are empirically closer to full fine-tuning (FFT) in terms of aggregation robustness ( 20 ) . Figure 3 supports this intuition: on synthetic landscapes, FedGaLore reaches flatter basins more often than LoRA, and on non-IID ViT-MNIST, it exhibits improved linear connectivity between client models, indicating enhanced aggregation stability. Algorithm 1 FedGaLore (one communication round k ; one adapted block \bm{W}\in\mathbb{R}^{n\times n} ) 0: Global model \bar{\theta}_{k} ; rank r ; local steps T ; projector refresh period \tau ; client weights \{\tilde{p}_{i}\} ; projection seed s_{k} . 1: Server: broadcast \bar{\theta}_{k} (and seed s_{k} if using seeded random projectors) to participating clients \mathcal{P}_{k} . 2: for all i\in\mathcal{P}_{k} in parallel do 3: Initialize \theta^{i,k}_{0}\leftarrow\bar{\theta}_{k} and GaLoreAdamW state for the adapted block: projected buffers (\tilde{\bm{m}}^{i,k}_{0},\tilde{\bm{v}}^{i,k}_{0})\in\mathbb{R}^{n\times r} and projector \bm{P}^{i,k}_{0} . 4: for t=0 to T-1 do 5: Compute gradient on the adapted block: \bm{g}^{i,k}_{t}=\nabla_{\bm{W}}\ell(\theta^{i,k}_{t};\mathcal{B}^{i,k}_{t})\in\mathbb{R}^{n\times n} . 6: GaLoreAdamW step (one-sided projection + buffer reprojection when refreshed every \tau steps): \theta^{i,k}_{t+1}\leftarrow\mathrm{GaLoreAdamW}(\theta^{i,k}_{t},\bm{g}^{i,k}_{t};\,r,\tau,s_{k}) . 7: end for 8: Client update: \Delta\theta_{i}\leftarrow\theta^{i,k}_{T}-\bar{\theta}_{k} (rank- r factorized). 9: Upload (\Delta\theta_{i},\tilde{\bm{v}}^{i,k}_{T}) , where \tilde{\bm{v}}^{i,k}_{T}\in\mathbb{R}^{n\times r} is the projected second-moment buffer for this block. 10: end for 11: Server: aggregate parameters (FedAvg): \bar{\theta}_{k+1}\leftarrow\bar{\theta}_{k}+\sum_{i\in\mathcal{P}_{k}}\tilde{p}_{i}\,\Delta\theta_{i} . 12: Construct matrix views using the shared seed/basis and synchronize second moments: \bar{\bm{v}}_{k+1}\leftarrow\mathrm{AJIVE}\big(\{\bm{V}^{i,k}\}_{i\in\mathcal{P}_{k}}\big) , where \bm{V}^{i,k}\coloneqq\tilde{\bm{v}}^{i,k}_{T}\,\bm{R}_{k}^{\top}\in\mathbb{R}^{n\times n} and \bm{R}_{k} is the (right) basis reconstructed from s_{k} . 13: Broadcast (\bar{\theta}_{k+1},\bar{\bm{v}}_{k+1}) (and next seed s_{k+1} if used). Server-side: FedAvg for parameters, AJIVE for second-moment synchronization. The server aggregates model parameters by FedAvg, \bar{\theta}_{k+1}\leftarrow\bar{\theta}_{k}+\sum_{i\in\mathcal{P}_{k}}\tilde{p}_{i}\,\Delta\theta_{i} . To mitigate optimizer-state mismatch with minimal communication, FedGaLore synchronizes only the projected second-moment buffer (the preconditioner state), which our W.H.P. analysis identifies as the most sensitive component under heterogeneity (Theorem 4.4 ). Concretely, for each adapted block, client i uploads its projected second-moment buffer \tilde{\bm{v}}^{i,k}_{T}\in\mathbb{R}^{n\times r} (together with the model update). Seeded matrix view. In the random-adaptive regime, the projector is determined by an integer seed s_{k} broadcast by the server; all parties deterministically reconstruct the same orthonormal basis \bm{R}_{k}\in\mathbb{R}^{n\times r} for each block. 3 3 3 For square blocks under the default right projection, \bm{R}_{k} is the right basis; rectangular blocks follow the side chosen by GaLore’s std rule. Then, for each uploaded \tilde{\bm{v}}^{i,k}_{T} server can project them back to the full space using: \bm{V}^{i,k}\;\coloneqq\;\tilde{\bm{v}}^{i,k}_{T}\,\bm{R}_{k}^{\top}\in\mathbb{R}^{n\times n}, (11) so each client contributes a rank- \leq r within the a shared space \mathbb{R}^{n\times n} . Why AJIVE. Under non-IID data, these views contain a shared component (task-level curvature/preconditioning structure) plus client-specific drift and noise. We model \bm{V}^{i,k}=\bm{J}^{k}+\bm{A}^{i,k}+\bm{E}^{i,k}, (12) and apply AJIVE ( 14 ) across \{\bm{V}^{i,k}\}_{i\in\mathcal{P}_{k}} to estimate the joint low-rank component \bm{J}^{k} (Details in Appendix E ). AJIVE uses a joint-rank parameter k (shared component rank); we set k=r so the synchronized state matches the client projector dimension and can be broadcast as the next-round initializer. Appendix F provides validation experiments showing AJIVE is more robust than naive averaging (even with post-hoc SVD re-projection). Computation and communication. Table 2 : Per-round overhead for one adapted block \bm{W}\in\mathbb{R}^{n\times n} . Client VRAM counts optimizer buffers for this block only (activations excluded). Uplink counts client \rightarrow server payload for this block. Federated LoRA FedGaLore Client VRAM (states) \mathcal{O}(nr) \mathcal{O}(nr) Client extra compute – \textsc{ProjRefresh}/\tau Uplink (per round) \mathcal{O}(nr) \mathcal{O}(nr) (update) +\ \mathcal{O}(nr) ( \tilde{\bm{v}} ) Server extra compute – \textsc{AJIVE}(K,n) (Table 7 ) We summarize client and server overhead for a single adapted block \bm{W}\in\mathbb{R}^{n\times n} with rank r in Table 2 . On the client, FedGaLore maintains GaLore-projected optimizer buffers of shape n\times r and refreshes the projector every \tau steps (randomized SVD/RSVD in early epochs, then seeded random projection in later epochs), which amortizes the projection cost over local training. On the server, AJIVE is applied once per round per adapted block and scales with the number of participating clients (views =|\mathcal{P}_{k}| ), not the total client population. Appendix F benchmarks AJIVE on dense n\times n views up to n{=}1024 (Table 7 ); for views =5 it adds \approx 22 ms on an A100 and \approx 93 ms on CPU, which is small relative to per-round training time. Communication remains LoRA-like: the only additional client uplink beyond Federated LoRA is an n\times r projected second-moment buffer per adapted block (plus a seed/index when using seeded random projectors), which is the same order as transmitting LoRA factors and remains far smaller than dense n\times n states. 6 Experimental Evaluation Table 3 : GLUE Benchmark Results. ✓ denotes IID. ✕ denotes Non-IID, \Delta indicates the difference. More results in Appendix I . Method CoLA Acc% SST Acc% MRPC Acc% QQP Acc% ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta FedAvg-Full 83.6 83.0 \downarrow 0.6 94.6 94.0 \downarrow 0.6 89.7 83.0 \downarrow 6.7 84.4 82.1 \downarrow 2.3 FedIT 71.2 65.4 \downarrow 5.8 88.1 83.7 \downarrow 4.4 82.3 76.9 \downarrow 5.4 78.9 72.4 \downarrow 6.5 FFA-LoRA 84.0 78.3 \downarrow 5.7 94.1 88.4 \downarrow 5.7 89.3 82.1 \downarrow 7.2 84.7 78.2 \downarrow 6.5 FLoRA 84.7 75.1 \downarrow 9.6 92.1 85.2 \downarrow 6.9 87.8 82.5 \downarrow 5.3 86.3 77.1 \downarrow 9.2 LoRA-Fair 81.7 80.1 \downarrow 1.6 87.4 84.2 \downarrow 3.2 86.5 82.5 \downarrow 4.0 83.9 82.1 \downarrow 1.8 FR-LoRA 83.9 79.4 \downarrow 4.5 93.7 87.6 \downarrow 6.1 84.3 79.2 \downarrow 5.1 88.2 80.1 \downarrow 8.1 FedGaLore - 83.2 81.4 \downarrow 1.8 94.5 90.2 \downarrow 4.3 89.8 84.5 \downarrow 5.3 85.4 81.5 \downarrow 3.9 FedGaLore 83.9 83.1 \downarrow 0.8 94.8 93.9 \downarrow 0.9 89.8 88.2 \downarrow 1.6 84.3 82.5 \downarrow 1.8 Table 4 : DomainNet Results (excluding Real domain). Accuracies in %. ✓ denotes IID, ✕ denotes Non-IID, \Delta indicates the difference. Method Clipart Painting Infograph Quickdraw Sketch ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta FedAvg-Full 82.3 78.9 \downarrow 3.4 79.2 75.8 \downarrow 3.4 54.3 48.5 \downarrow 5.8 71.0 67.2 \downarrow 3.8 78.3 74.1 \downarrow 4.2 FedIT 80.1 74.2 \downarrow 5.9 77.5 70.8 \downarrow 6.7 52.8 45.9 \downarrow 6.9 69.4 62.8 \downarrow 6.6 76.2 69.1 \downarrow 7.1 FFA-LoRA 79.8 72.5 \downarrow 6.3 76.9 71.2 \downarrow 5.7 51.9 46.2 \downarrow 5.7 68.7 63.5 \downarrow 5.2 75.8 68.4 \downarrow 7.4 FLoRA 80.7 74.1 \downarrow 6.6 77.8 71.3 \downarrow 6.5 52.1 45.0 \downarrow 7.1 69.5 63.2 \downarrow 6.3 76.0 68.5 \downarrow 7.5 LoRA-Fair 82.3 81.1 \downarrow 1.2 73.2 70.3 \downarrow 2.9 43.8 41.0 \downarrow 2.8 63.3 59.1 \downarrow 4.2 73.4 68.5 \downarrow 4.9 FR-LoRA 81.7 77.9 \downarrow 3.8 78.2 74.1 \downarrow 4.1 52.1 43.3 \downarrow 8.8 70.2 64.6 \downarrow 5.6 77.1 69.1 \downarrow 8.0 FedGaLore - 82.1 71.3 \downarrow 10.8 79.5 72.4 \downarrow 7.1 53.8 41.9 \downarrow 11.9 69.2 64.3 \downarrow 4.9 76.8 70.5 \downarrow 6.3 FedGaLore 81.9 78.2 \downarrow 3.7 78.8 75.1 \downarrow 3.7 53.8 47.6 \downarrow 6.2 70.5 66.4 \downarrow 4.1 77.9 73.5 \downarrow 4.4 We evaluate FedGaLore on three domains: NLU: RoBERTa-base ( 36 ) on 7 GLUE tasks ( 56 ) ; Vision: ViT-base ( 11 ) on 6 DomainNet domains ( 44 ) ; NLG: Llama-2-7B ( 55 ) fine-tuned on MetaMathQA ( 68 ) and evaluated on GSM8K ( 9 ) and MATH ( 21 ) . Baselines. We compare to FedAvg-Full (full fine-tuning 4 4 4 For Llama-2-7B, “full fine-tuning” refers to updating the full weight matrices of the target modules where LoRA/GaLore is applied (not the entire model). ), FedIT ( 70 ) , FFA-LoRA ( 52 ) , FLoRA ( 60 ) , FR-LoRA ( 64 ) , and LoRA-Fair ( 4 ) . We use each baseline’s original optimizer choice when available, and otherwise match learning rate, rank, and target modules. We also report FedGaLore - (GaLore in FL without drift-robust state synchronization) as an ablation. All PEFT methods use the same target modules and rank. Federated protocol. For NLU and vision, we use M{=}50 clients with partial participation ( K{=}5 /round, sampled uniformly). For NLG, we use M{=}4 clients with full participation ( K{=}4 ) due to higher per-client cost. All methods use the same training budget (rounds and local steps). Hyperparameters and implementation details are in Appendix G . Table 5 : LLM Federated Fine-Tuning Results. Accuracies in % on GSM8K and Math. ✓ denotes IID, ✕ denotes Non-IID, \Delta indicates the difference. Higher is better. Method GSM8K MATH ✓ ✕ \Delta ✓ ✕ \Delta FedAvg-Full 35.0 32.3 \downarrow 2.7 4.5 4.1 \downarrow 0.4 FedIT 31.3 28.2 \downarrow 3.1 4.7 4.0 \downarrow 0.7 FFA-LoRA 28.0 26.2 \downarrow 1.8 3.8 3.8 \downarrow 0.0 LoRA-Fair 29.3 28.5 \downarrow 0.8 4.1 3.8 \downarrow 0.3 FLoRA 39.8 37.2 \downarrow 2.6 4.6 4.0 \downarrow 0.6 FR-LoRA 44.3 39.2 \downarrow 5.1 5.2 4.3 \downarrow 0.9 FedGaLore - 41.7 38.5 \downarrow 3.2 5.6 4.9 \downarrow 0.7 FedGaLore 44.9 41.9 \downarrow 2.7 5.7 5.1 \downarrow 0.6 Results and discussion. We report IID and non-IID ( \alpha=0.5 ) performance and use their gap as a simple sensitivity metric: smaller \Delta indicates lower sensitivity to non-IIDness. NLU (GLUE) and vision (DomainNet). Across GLUE (Table 3 ) and DomainNet (Table 4 ), federated LoRA baselines generally show much larger robustness gaps than FedAvg-Full, whose minimal degradation suggests that full-space optimization/aggregation can remain stable under these shifts. Several LoRA variants incur large drops (e.g., CoLA and multiple DomainNet domains), consistent with update-space mismatch : clients adapt in restricted low-rank directions that become misaligned across clients, and aggregation amplifies off-subspace components. Methods that constrain the update space more aggressively (e.g., FFA-LoRA, LoRA-Fair) reduce \Delta in some settings but at the cost of lower IID accuracy, indicating reduced expressivity. FedGaLore improves this trade-off, matching or improving IID performance while maintaining consistently small \Delta across tasks/domains. The ablation FedGaLore - boosts IID accuracy but collapses under data heterogeneity (notably on DomainNet), showing that dynamic subspaces alone are insufficient— drift-robust state synchronization is essential. NLG (LLM fine-tuning). Robustness gaps are smaller on GSM8K and MATH (Table 5 ), suggesting the pretrained Llama-2 backbone provides stronger invariances. Local-adaptive methods (FLoRA/FR-LoRA/ FedGaLore - ) also achieve higher IID performance than static LoRA baselines, indicating that reasoning tasks benefit from greater optimization plasticity. However, local adaptivity increases sensitivity when optimizer states drift across clients. FedGaLore retains strong IID performance while reducing the non-IID drop, consistent with combining gradient-subspace exploration with drift-robust second-moment synchronization. Practical takeaway. FedAvg-Full is the most robust but costly. Federated LoRA is efficient, but can be brittle. FedGaLore bridges this gap, delivering near-FFT robustness with PEFT-level efficiency, especially when local adaptivity is needed. 7 Conclusions and Future Work We present FedGaLore , a robust protocol for federated parameter-efficient fine-tuning under non-IID data. Our W.H.P. analysis attributes federated LoRA brittleness to two coupled mismatches—update-space and optimizer-state—and motivates combining GaLore-style gradient-subspace optimization with drift-robust synchronization of projected second-moment statistics, delivering consistent gains across NLU, vision, and LLM benchmarks. Beyond improved accuracy, FedGaLore yields a practical stability–efficiency trade-off: near full fine-tuning robustness without transmitting dense optimizer states, enabled by low-rank projected synchronization and lightweight server-side filtering. Future work includes extending the analysis to broader participation regimes and develop a tighter end-to-end W.H.P. guarantees. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Details of Local Training Operator \mathcal{T} Algorithm 2 Local Training Operator \mathcal{T} (Stochastic Gradient Descent) 1: Input: global model \bar{\theta}^{k} , steps T , stepsize \eta 2: Output: local model \theta^{i,k}_{T} 3: Initialize \theta^{i,k}_{0}\leftarrow\bar{\theta}^{k} 4: for t=0 to T-1 do 5: Sample minibatch \mathcal{B}^{i,k}_{t}\sim\mathcal{D}_{i} 6: Compute stochastic gradient g^{i,k}_{t}=\nabla\ell(\theta^{i,k}_{t};\,\mathcal{B}^{i,k}_{t}) 7: Model update: \theta^{i,k}_{t+1}=\theta^{i,k}_{t}-\eta\,g^{i,k}_{t} 8: end for 9: Send \theta^{i,k}_{T} to the server Algorithm 3 Local Training Operator \mathcal{T} (SGD with Momentum) 1: Input: global model \bar{\theta}^{k} , local optimizer state S^{i,k}_{0} , steps T , stepsize \eta , momentum factor \mu 2: Output: local model \theta^{i,k}_{T} , local state S^{i,k}_{T} 3: Initialize \theta^{i,k}_{0}\leftarrow\bar{\theta}^{k} 4: Initialize momentum buffer v^{i,k}_{0}\leftarrow S^{i,k}_{0} {Initialize to \bm{0} if k=0 } 5: for t=0 to T-1 do 6: Sample minibatch \mathcal{B}^{i,k}_{t}\sim\mathcal{D}_{i} 7: Compute stochastic gradient g^{i,k}_{t}=\nabla\ell(\theta^{i,k}_{t};\,\mathcal{B}^{i,k}_{t}) 8: Update momentum buffer: v^{i,k}_{t+1}=\mu\,v^{i,k}_{t}+g^{i,k}_{t} 9: Model update: \theta^{i,k}_{t+1}=\theta^{i,k}_{t}-\eta\,v^{i,k}_{t+1} 10: end for 11: Set S^{i,k}_{T}=v^{i,k}_{T} 12: Send (\theta^{i,k}_{T},\ S^{i,k}_{T}) to the server Algorithm 4 Local Training Operator \mathcal{T} (Adam ( 25 ) / AdamW ( 38 ) ) 1: Input: global model \bar{\theta}^{k} , local optimizer state S^{i,k}_{0} , steps T , stepsize \eta , optimizer \mathcal{O} , momentum parameters (\beta_{1},\beta_{2}) , numerical constant \varepsilon 2: Output: local model \theta^{i,k}_{T} , local state S^{i,k}_{T} 3: Initialize \theta^{i,k}_{0}\leftarrow\bar{\theta}^{k} 4: Initialize first and second momentum m^{i,k}_{0},v^{i,k}_{0}\leftarrow S^{i,k}_{0} 5: for t=0 to T-1 do 6: Sample minibatch \mathcal{B}^{i,k}_{t}\sim\mathcal{D}_{i} 7: Compute stochastic gradient g^{i,k}_{t}=\nabla\ell(\theta^{i,k}_{t};\,\mathcal{B}^{i,k}_{t}) 8: Update first momentum: m^{i,k}_{t+1}=\beta_{1}m^{i,k}_{t}+(1-\beta_{1})g^{i,k}_{t} 9: Update second moment: v^{i,k}_{t+1}=\beta_{2}v^{i,k}_{t}+(1-\beta_{2})\,g^{i,k}_{t}\odot g^{i,k}_{t} 10: Model update: \theta^{i,k}_{t+1}=\theta^{i,k}_{t}-\eta\frac{m^{i,k}_{t+1}}{\sqrt{v^{i,k}_{t+1}}+\varepsilon} 11: end for 12: Set S^{i,k}_{T}=(m^{i,k}_{T},v^{i,k}_{T}) ; 13: Send (\theta^{i,k}_{T},\ S^{i,k}_{T}) to the server A.1 Details of GaLore-AdamW This section summarizes the GaLoreAdamW update used by FedGaLore and clarifies the tensor shapes under the default setting ( proj_type=std , one-sided projection) for an adapted weight block \bm{W}\in\mathbb{R}^{m\times n} (square case m{=}n is a special case). Default projection rule and shapes. GaLore maintains a rank- r projector that is refreshed every update_proj_gap steps (denoted \tau in the paper). Under proj_type=std , the projector chooses the side based on the aspect ratio: • If m\geq n (including square blocks), use a right basis \bm{P}\in\mathbb{R}^{r\times n} and project \tilde{\bm{g}}=\bm{g}\bm{P}^{\top}\in\mathbb{R}^{m\times r} . • If m0 ; (ii) bounded heterogeneity holds: \frac{1}{M}\sum_{i=1}^{M}\|\nabla F_{i}(\theta)\|_{2}^{2}\leq H^{2}+B^{2}\|\nabla f(\theta)\|_{2}^{2},\quad\forall\theta\in Q; (iii) mini-batch noise is unbiased with bounded conditional second moment: \mathbb{E}[\xi_{t}^{i,k}\mid\mathcal{F}_{k,t-1}]=0 and \mathbb{E}[\|\xi_{t}^{i,k}\|_{2}^{2}\mid\mathcal{F}_{k,t-1}]\leq\sigma^{2} ; and (iv) \|\nabla f(\theta)\|_{2}\leq G on Q . Let the within-round reference be T steps of (full-batch) gradient descent on f : \theta_{t+1}^{\star,k}=\theta_{t}^{\star,k}-\eta\nabla f(\theta_{t}^{\star,k}) with \theta_{0}^{\star,k}=\bar{\theta}^{k} . If \eta LT\leq\frac{1}{6} , then for each round k , \mathbb{E}\Big[\|\bar{\theta}^{k+1}-\theta_{T}^{\star,k}\|_{2}\ \big|\ \bar{\theta}^{k}\Big]\;\leq\;2\eta T\left(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\right). Moreover, defining r_{k}\coloneqq\mathbb{E}[f(\bar{\theta}^{k})-f^{\star}] , there exist absolute constants c_{1},c_{2}>0 such that r_{k+1}\leq(1-\eta\mu)^{T}\,r_{k}+c_{1}\,\eta T\left(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\right)+c_{2}\,\eta^{2}T^{2}\left(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\right)^{2}. In particular, if H=\sigma=0 (and hence clients are effectively IID and gradients are deterministic), then r_{k} converges linearly to 0 (i.e., \bar{\theta}^{k} converges in expectation to f^{\star} ). More generally, with constant \eta the recursion converges linearly to a neighborhood whose radius scales with H and \sigma , and with diminishing \eta\to 0 the neighborhood vanishes. Proof. Fix a round k and omit the superscript k for readability. Let \Delta_{t}^{i}\coloneqq\theta_{t}^{i}-\theta_{t}^{\star} and define the RMS deviation D_{t}\coloneqq\left(\frac{1}{M}\sum_{i=1}^{M}\|\Delta_{t}^{i}\|_{2}^{2}\right)^{1/2},\qquad D_{0}=0. Step 1: an RMS recursion using bounded heterogeneity. For SGD, \theta_{t+1}^{i}=\theta_{t}^{i}-\eta(\nabla F_{i}(\theta_{t}^{i})+\xi_{t}^{i}) and \theta_{t+1}^{\star}=\theta_{t}^{\star}-\eta\nabla f(\theta_{t}^{\star}) , so \Delta_{t+1}^{i}=\Delta_{t}^{i}-\eta\big(\nabla f(\theta_{t}^{i})-\nabla f(\theta_{t}^{\star})\big)-\eta\big(\nabla F_{i}(\theta_{t}^{i})-\nabla f(\theta_{t}^{i})\big)-\eta\xi_{t}^{i}. Stack the vectors across clients: \widetilde{\Delta}_{t}\in\mathbb{R}^{Md} collects (\Delta_{t}^{1},\dots,\Delta_{t}^{M}) , and similarly define stacked \widetilde{A}_{t} for (\nabla f(\theta_{t}^{i})-\nabla f(\theta_{t}^{\star}))_{i=1}^{M} , \widetilde{B}_{t} for (\nabla F_{i}(\theta_{t}^{i})-\nabla f(\theta_{t}^{i}))_{i=1}^{M} , and \widetilde{\xi}_{t} for (\xi_{t}^{i})_{i=1}^{M} . Then \widetilde{\Delta}_{t+1}=\widetilde{\Delta}_{t}-\eta\widetilde{A}_{t}-\eta\widetilde{B}_{t}-\eta\widetilde{\xi}_{t} and by the triangle inequality in \mathbb{R}^{Md} , \|\widetilde{\Delta}_{t+1}\|_{2}\leq\|\widetilde{\Delta}_{t}-\eta\widetilde{A}_{t}\|_{2}+\eta\|\widetilde{B}_{t}\|_{2}+\eta\|\widetilde{\xi}_{t}\|_{2}. (27) Since f is L -smooth, \|\nabla f(\theta_{t}^{i})-\nabla f(\theta_{t}^{\star})\|_{2}\leq L\|\Delta_{t}^{i}\|_{2} for each i , hence \|\widetilde{A}_{t}\|_{2}\leq L\|\widetilde{\Delta}_{t}\|_{2} , and therefore \|\widetilde{\Delta}_{t}-\eta\widetilde{A}_{t}\|_{2}\leq(1+\eta L)\|\widetilde{\Delta}_{t}\|_{2} . Dividing equation 27 by \sqrt{M} gives D_{t+1}\leq(1+\eta L)D_{t}+\eta\,\underbrace{\left(\frac{1}{M}\sum_{i=1}^{M}\|\nabla F_{i}(\theta_{t}^{i})-\nabla f(\theta_{t}^{i})\|_{2}^{2}\right)^{1/2}}_{=:B_{t}}+\eta\,\underbrace{\left(\frac{1}{M}\sum_{i=1}^{M}\|\xi_{t}^{i}\|_{2}^{2}\right)^{1/2}}_{=:N_{t}}. (28) We bound B_{t} by relating drift at \theta_{t}^{i} to drift at the common reference point \theta_{t}^{\star} : \nabla F_{i}(\theta_{t}^{i})-\nabla f(\theta_{t}^{i})=\underbrace{\nabla F_{i}(\theta_{t}^{\star})-\nabla f(\theta_{t}^{\star})}_{\delta_{i}(\theta_{t}^{\star})}+\big(\nabla F_{i}(\theta_{t}^{i})-\nabla F_{i}(\theta_{t}^{\star})\big)+\big(\nabla f(\theta_{t}^{\star})-\nabla f(\theta_{t}^{i})\big). Taking RMS over i and using the triangle inequality plus L -smoothness of F_{i} and f yields B_{t}\leq\left(\frac{1}{M}\sum_{i=1}^{M}\|\delta_{i}(\theta_{t}^{\star})\|_{2}^{2}\right)^{1/2}+2LD_{t}. (29) Now compute the RMS drift at a common point using bounded heterogeneity: \frac{1}{M}\sum_{i=1}^{M}\|\delta_{i}(\theta)\|_{2}^{2}=\frac{1}{M}\sum_{i=1}^{M}\|\nabla F_{i}(\theta)\|_{2}^{2}-\|\nabla f(\theta)\|_{2}^{2}\leq H^{2}+(B^{2}-1)\|\nabla f(\theta)\|_{2}^{2}\leq H^{2}+(B^{2}-1)G^{2}. Therefore \left(\frac{1}{M}\sum_{i}\|\delta_{i}(\theta_{t}^{\star})\|_{2}^{2}\right)^{1/2}\leq\sqrt{H^{2}+(B^{2}-1)G^{2}} . For the noise term, by Jensen and the conditional variance bound, \mathbb{E}[N_{t}\mid\mathcal{F}_{t-1}]\leq\sqrt{\mathbb{E}[N_{t}^{2}\mid\mathcal{F}_{t-1}]}=\sqrt{\frac{1}{M}\sum_{i=1}^{M}\mathbb{E}[\|\xi_{t}^{i}\|_{2}^{2}\mid\mathcal{F}_{t-1}]}\leq\sigma. Taking conditional expectations in equation 28 and using equation 29 gives \mathbb{E}[D_{t+1}\mid\mathcal{F}_{t-1}]\leq(1+\eta L)D_{t}+\eta\big(\sqrt{H^{2}+(B^{2}-1)G^{2}}+2LD_{t}\big)+\eta\sigma=(1+3\eta L)D_{t}+\eta\sqrt{H^{2}+(B^{2}-1)G^{2}}+\eta\sigma. Step 2: unrolling the RMS recursion. Iterating from D_{0}=0 yields \mathbb{E}[D_{T}\mid\bar{\theta}^{k}]\leq\eta\sum_{t=0}^{T-1}(1+3\eta L)^{T-1-t}\Big(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\Big)\leq\eta T\,(1+3\eta L)^{T}\Big(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\Big). Under \eta LT\leq\frac{1}{6} , we have (1+3\eta L)^{T}\leq e^{3\eta LT}\leq e^{1/2}\leq 2 , hence \mathbb{E}[D_{T}\mid\bar{\theta}^{k}]\leq 2\eta T\Big(\sqrt{H^{2}+(B^{2}-1)G^{2}}+\sigma\Big). Since \bar{\theta}^{k+1}-\theta_{T}^{\star}=\frac{1}{M}\sum_{i}(\theta_{T}^{i}-\theta_{T}^{\star}) , we have \|\bar{\theta}^{k+1}-\theta_{T}^{\star}\|_{2}\leq\frac{1}{M}\sum_{i}\|\Delta_{T}^{i}\|_{2}\leq D_{T} . This proves the first inequality in the corollary. Step 3: transferring reference contraction to the global iterate. By Assumption 3.6 and Assumption 3.5 , gradient descent on f with step size \eta\leq 1/L satisfies the standard linear rate f(\theta_{T}^{\star})-f^{\star}\leq(1-\eta\mu)^{T}\big(f(\bar{\theta}^{k})-f^{\star}\big). Next, by L -smoothness of f , f(\bar{\theta}^{k+1})\leq f(\theta_{T}^{\star})+\langle\nabla f(\theta_{T}^{\star}),\bar{\theta}^{k+1}-\theta_{T}^{\star}\rangle+\frac{L}{2}\|\bar{\theta}^{k+1}-\theta_{T}^{\star}\|_{2}^{2}. Using \|\nabla f(\theta)\|_{2}\leq G on Q and e_{k}\coloneqq\bar{\theta}^{k+1}-\theta_{T}^{\star} , f(\bar{\theta}^{k+1})-f^{\star}\leq f(\theta_{T}^{\star})-f^{\star}+G\|e_{k}\|_{2}+\frac{L}{2}\|e_{k}\|_{2}^{2}. Taking expectations and using \|e_{k}\|\leq D_{T} and the bound on \mathbb{E}[D_{T}\mid\bar{\theta}^{k}] yields r_{k+1}\leq(1-\eta\mu)^{T}r_{k}+G\,\mathbb{E}[D_{T}]+\frac{L}{2}\mathbb{E}[D_{T}^{2}]. Finally, use Jensen to bound \mathbb{E}[D_{T}^{2}]\leq(\mathbb{E}[D_{T}])^{2} and substitute the bound on \mathbb{E}[D_{T}] . This gives the claimed recursion with explicit constants (absorbed into c_{1},c_{2} ). The special cases ( H=\sigma=0 and/or diminishing stepsizes) follow immediately. ∎ Appendix C Details of Synthetic Loss Landscape Experiments To empirically validate our intuition that GaLore tends to find solutions with better loss landscape geometry, we designed a controlled synthetic experiment. The goal is to isolate how the dynamics of low-rank optimizers influence their ability to escape ”sharp, aligned” local minima in favor of ”flat, orthogonal” global basins. C.1 Why Flatness implies Aggregation Robustness In Federated Learning, the geometric property of “flatness” is a direct proxy for aggregability. We seek a global model \bm{W}^{\star} that remains valid even when averaged with perturbed local models. If client models drift by \Delta\bm{W}_{i} , the expected loss of the aggregate is approximately: \mathbb{E}[\mathcal{L}(\bm{W}_{\mathrm{agg}})]\approx\mathcal{L}(\bm{W}^{\star})+\frac{1}{2}\text{Tr}\left(\nabla^{2}\mathcal{L}(\bm{W}^{\star})\mathbb{E}[\Delta\bm{W}\Delta\bm{W}^{\top}]\right). (30) In a sharp valley , the Hessian \nabla^{2}\mathcal{L} has large eigenvalues, so even small drift incurs a massive penalty. In a flat basin , eigenvalues are small, minimizing the impact of heterogeneity. C.2 Landscape Construction: The ”Kinetic Trap” We optimize a matrix \bm{W}\in\mathbb{R}^{d\times d} . The landscape combines two quadratic basins via a smooth SoftMin function: \mathcal{L}(\bm{W})=-\tau\log\left(\exp(-\mathcal{L}_{1}(\bm{W})/\tau)+\exp(-\mathcal{L}_{2}(\bm{W})/\tau)\right). (31) We define two key directions relative to the initialization \bm{W}_{0} : • \mathbf{e}_{1} ( Aligned ): A direction lying within the initial LoRA subspace span( \bm{A}_{0}^{\top} ). • \mathbf{e}_{2} ( Orthogonal ): A direction orthogonal to the initialization. The basins are constructed to create a specific geometric challenge: • Basin 1 (The Target): A flat basin centered at \bm{W}_{1}=c\cdot\mathbf{e}_{2} . This minimum is robust (small eigenvalues in all directions) but requires significant rotation into the orthogonal direction \mathbf{e}_{2} to be discovered. • Basin 2 (The Trap): A sharp valley centered at the origin but elongated along \mathbf{e}_{1} . We set the Hessian \bm{H}_{2} such that the valley is steep in orthogonal directions but shallow along \mathbf{e}_{1} . This creates a “trap” : LoRA’s dynamics, which prioritize movement in the current subspace, perceive a quick path of descent into this brittle minimum. C.3 Optimization Setup We compare three methods, initializing each trial from a randomized starting point \bm{W}^{(0)}=\bm{W}_{\mathrm{ref}}+\bm{\Xi} , where \bm{W}_{\mathrm{ref}} is a fixed reference point between the two basins and \bm{\Xi} is Gaussian noise drawn for each trial. 1. Full-Space SGD: Updates \bm{W}_{t+1}\leftarrow\bm{W}_{t}-\alpha\nabla\mathcal{L} . 2. LoRA: Updates \bm{W}=\bm{W}_{0}+\bm{B}\bm{A} . Although the subspace is not strictly fixed, the coupling between \bm{B} and \bm{A} creates an “inertial alignment” that resists rapid rotation into \mathbf{e}_{2} . 3. GaLore: Explicitly projects gradients onto a time-varying subspace \bm{U}_{r}\bm{\Sigma}_{r}\bm{V}_{r}^{\top} derived from the SVD of the gradient, allowing for active subspace rotation. C.4 Results Table 6 confirms that LoRA is attracted to the ”aligned” sharp valley because its natural dynamics are insufficient to traverse the orthogonal gap. GaLore’s active rotation allows it to escape the trap and converge to the flat basin, matching the robustness of Full SGD. Table 6 : Probability of converging to the Flat Basin (Robust) vs. Sharp Valley (Brittle). Method Basin 1 (Flat/Robust) Basin 2 (Sharp/Brittle) Mechanism Full SGD 91% 9% Unconstrained GaLore 60% 40% Active Rotation LoRA 20% 80% Inertial Alignment Appendix D SVD-to-random projectors: efficiency and local convergence Motivation. GaLore-style methods require periodically refreshing a rank- r projector. A purely data-driven refresh (SVD/RSVD every \tau steps) improves early optimization but incurs nontrivial overhead, while purely random projection has lower projector-estimation overhead but may start from a poorly aligned subspace. Following prior work on unbiased/efficient GaLore variants ( 42 ) , we adopt an SVD-to-random schedule: In our implementation, we use SVD/RSVD for the first S projector refreshes (set by adaptive_proj_steps ) and then switch to seeded random projectors thereafter. In the random phase, the projector is fully determined by an integer seed, so we broadcast only the seed (not the basis matrix), enabling deterministic reconstruction across clients. Variants. We compare three projector schedules: (i) FedGaLore (ours): RSVD/SVD in early epochs \rightarrow seeded random thereafter; (ii) FedGaLore-SVD : always RSVD/SVD refresh; (iii) FedGaLore-Pure Random : always seeded random projector. We also plot representative federated LoRA baselines (FFA-LoRA-AdamW, FLoRA-AdamW) for reference. (a) RoBERTa (GLUE): local training loss vs wall-clock time. (b) ViT (DomainNet): local training loss vs wall-clock time. Figure 4 : Local convergence under different projector schedules. FedGaLore (SVD \rightarrow random) achieves the best time-to-loss compared to always-SVD and always-random schedules. Results and discussion (time-to-loss, same steps). Figure 4 reports local training loss versus wall-clock time under a fixed local training budget: all variants run the same number of local steps with identical hyperparameters (rank, \tau , learning rate, batch size, etc.). Differences therefore reflect time-per-step (including projector refresh/reprojection overhead) and optimization efficiency per step, rather than additional steps. Across both RoBERTa and ViT, FedGaLore (SVD \rightarrow random) achieves the best time-to-loss. Compared to FedGaLore-SVD , this is expected since repeatedly refreshing RSVD/SVD bases and reprojecting buffers incurs nontrivial overhead. FedGaLore-Pure Random is consistently worse in time-to-loss, indicating that fully random projectors can slow local optimization in practice. While the exact wall-clock advantage over pure random can depend on implementation details (e.g., orthonormalization kernels, caching, and refresh schedules), the empirical takeaway is clear: a short data-driven warm start followed by seeded random projectors yields the most favorable time-to-loss. Comparison to federated LoRA baselines. Figure 4 also includes representative LoRA baselines under the same local-step budget. We observe that LoRA with AdamW can achieve fast local loss reduction, consistent with the benefit of local adaptivity. In contrast, FFA-LoRA is typically slower locally in these settings which is consistent with its more restrictive update parameterization that trades expressivity for stability under heterogeneity. Notably, these local curves do not capture global robustness: our main results show that methods with strong local progress can still suffer from optimizer-state and update-space mismatch under non-IID aggregation, motivating FedGaLore ’s server-side state synchronization. Appendix E Angle-based Joint and Individual Variation Explained (AJIVE) Table 7 : AJIVE scalability microbenchmark (server-side). We measure AJIVE latency as a function of the number of views (participating clients per round, views =|\mathcal{P}_{k}| ) and matrix size for dense square inputs v\in\mathbb{R}^{n\times n} (avg over trials). GPU uses an A100 with randomized SVD. FLOPs are estimates. Device views ( |\mathcal{P}_{k}| ) n Average time (s) Average FLOPs (est.) CPU 1 512 0.006554 4.375e+06 CPU 1 768 0.009816 9.708e+06 CPU 1 1024 0.018628 1.714e+07 CPU 2 512 0.012233 8.849e+06 CPU 2 768 0.025425 1.956e+07 CPU 2 1024 0.040607 3.447e+07 CPU 5 512 0.031390 2.193e+07 CPU 5 768 0.045956 4.862e+07 CPU 5 1024 0.092544 8.579e+07 CPU 10 512 0.060522 4.372e+07 CPU 10 768 0.098389 9.704e+07 CPU 10 1024 0.184439 1.713e+08 GPU (A100) 1 512 0.031459 4.375e+06 GPU (A100) 1 768 0.004896 9.708e+06 GPU (A100) 1 1024 0.004826 1.714e+07 GPU (A100) 1 2 512 0.858201 8.849e+06 GPU (A100) 2 768 0.009561 1.956e+07 GPU (A100) 2 1024 0.009093 3.447e+07 GPU (A100) 5 512 0.867758 2.193e+07 GPU (A100) 5 768 0.023258 4.862e+07 GPU (A100) 5 1024 0.022056 8.579e+07 GPU (A100) 10 512 0.747936 4.372e+07 GPU (A100) 10 768 0.046743 9.704e+07 GPU (A100) 10 1024 0.045271 1.713e+08 1 GPU timings include synchronization and may exhibit variance due to kernel launch/allocator effects. This may lead to abnormal execution time. We provide a brief introduction to the Joint and Individual Variation Explained (JIVE) framework and the Angle-based JIVE (AJIVE) method. The decomposition of multi-view data into shared and specific components is a fundamental challenge that has sparked diverse applications, such as identifying communities within heterogeneous networks ( 39 ) , analyzing high-dimensional genomic data ( 37 ) , and discerning global and local features in federated learning ( 49 ) . A significant challenge in these analyses lies in separating the joint and individual variations present in multi-view datasets. In their seminal work, 37 pioneered the JIVE model to address this decomposition. To overcome computational bottlenecks and identifiability issues inherent in the original formulation, 14 proposed AJIVE, a two-stage spectral method based on principal angle analysis. Recent theoretical work by 66 establishes that AJIVE achieves optimal performance in high-signal-to-noise ratio (SNR) settings, confirming its efficiency while outlining specific limitations; we refer interested readers to this work for details. Our implementation follows the logic of the MVLearn library ( 45 ) . For computational efficiency, we have reimplemented the method using PyTorch ( 43 ) to leverage GPU acceleration. Algorithm 5 details the AJIVE procedure. Algorithm 5 Angle-based Joint and Individual Variation Explained (AJIVE) 1: Input: Data matrices \{\bm{X}^{(1)},\dots,\bm{X}^{(k)}\} , parameters for rank estimation 2: Output: Joint matrices \bm{J}^{(i)} , Individual matrices \bm{I}^{(i)} , Noise matrices \bm{E}^{(i)} 3: // Phase 1: Signal Space Initial Extraction 4: for i=1 to k do 5: Center \bm{X}^{(i)} by subtracting column means 6: Estimate initial signal rank r_{\text{init}}^{(i)} 7: Compute SVD: \bm{U}^{(i)},\bm{D}^{(i)},\bm{V}^{(i)}\leftarrow\text{SVD}(\bm{X}^{(i)},\text{rank}=r_{\text{init}}^{(i)}) 8: Calculate SV threshold \tau^{(i)} (e.g., midpoint of r and r+1 SVs) 9: end for 10: // Phase 2: Score Space Segmentation 11: Concatenate signal scores: \bm{M}\leftarrow[\bm{U}^{(1)},\bm{U}^{(2)},\dots,\bm{U}^{(k)}] 12: Compute Joint SVD: \bm{U}_{\text{joint}},\bm{D}_{\text{joint}},\bm{V}_{\text{joint}}\leftarrow\text{SVD}(\bm{M}) 13: if Joint Rank r_{\text{joint}} is not provided then 14: Compute Wedin bound \delta_{\text{wedin}} and Random bound \delta_{\text{rand}} via resampling 15: Set cutoff \delta_{\text{cutoff}}\leftarrow\max(\delta_{\text{wedin}},\delta_{\text{rand}}) 16: r_{\text{joint}}\leftarrow\text{count}(\bm{D}_{\text{joint}}^{2}>\delta_{\text{cutoff}}) 17: end if 18: Extract basis: \bm{U}_{\text{joint}}\leftarrow\bm{U}_{\text{joint}}[:,1:r_{\text{joint}}] 19: Remove columns from \bm{U}_{\text{joint}} if projection norm on \bm{X}^{(i)}<\tau^{(i)} (Identifiability Check) 20: // Phase 3: Final Decomposition 21: for i=1 to k do 22: Joint Estimate: \bm{J}^{(i)}\leftarrow\bm{U}_{\text{joint}}\bm{U}_{\text{joint}}^{T}\bm{X}^{(i)} 23: Individual Estimate: 24: Compute residual: \bm{X}^{(i)}_{\text{orth}}\leftarrow\bm{X}^{(i)}-\bm{J}^{(i)} 25: Compute SVD: \bm{U}_{\text{ind}},\bm{D}_{\text{ind}},\bm{V}_{\text{ind}}\leftarrow\text{SVD}(\bm{X}^{(i)}_{\text{orth}}) 26: Determine rank r_{\text{ind}}^{(i)} via threshold \tau^{(i)} 27: \bm{I}^{(i)}\leftarrow\bm{U}_{\text{ind}}[:,:r_{\text{ind}}^{(i)}]\cdot\bm{D}_{\text{ind}}[:r_{\text{ind}}^{(i)}]\cdot\bm{V}_{\text{ind}}[:,:r_{\text{ind}}^{(i)}]^{T} 28: Noise Estimate: \bm{E}^{(i)}\leftarrow\bm{X}^{(i)}-\bm{J}^{(i)}-\bm{I}^{(i)} 29: end for 30: Return \{\bm{J}^{(i)},\bm{I}^{(i)},\bm{E}^{(i)}\}_{i=1}^{k} Appendix F Validation of Spectral Synchronization via AJIVE Figure 5 : AJIVE recovers global geometry under structured drift. We compare the reconstruction error of the global second moment V^{\star} ( y -axis) vs. number of clients ( x -axis). Naive Averaging (Red) fails to converge to the true signal because the squared drift terms \mathbb{E}[\bm{L}_{k}^{2}] introduce a persistent bias. SVD on Average (Solid lines) helps slightly but operates on already-corrupted data. AJIVE (Markers) successfully isolates the shared signal structure. Notably, AJIVE Rank-15 (Green circles) achieves the lowest error, correctly capturing the rank expansion induced by the element-wise square operation. To justify our design choice of using AJIVE for synchronizing the second-moment estimator v , we conduct a controlled synthetic experiment. The core challenge in federated adaptive optimization is that the second moment is a non-linear transformation of the gradients ( v\approx g^{2} ). Even if client gradients share a low-rank subspace, the client drift term c_{i} introduces structured noise that does not vanish under simple averaging due to the quadratic nature of the update. F.1 Experimental Setup We simulate a scenario where the true global gradient signal lies in a low-rank subspace, but clients observe a drifted, noisy version. We aim to recover the “true” global second moment \bm{V}^{\star} . Data Generation. Let the global gradient signal be \bm{G}^{\star}\in\mathbb{R}^{N\times M} with rank r=5 . The ground truth second moment is \bm{V}^{\star}=(\bm{G}^{\star})^{\odot 2} (element-wise square). Note that while \bm{G}^{\star} is rank-5, its element-wise square \bm{V}^{\star} typically has a higher rank (approximated by r(r+1)/2\approx 15 ). We simulate K clients. Client k observes a perturbed gradient: \bm{G}_{k}=\bm{G}^{\star}+\underbrace{\bm{L}_{k}}_{\text{Drift}}+\underbrace{\bm{\Xi}_{k}}_{\text{Noise}}, (32) where \bm{L}_{k} is a random rank-2 perturbation representing client-specific drift, and \bm{\Xi}_{k}\sim\mathcal{N}(0,\sigma^{2}) is unstructured noise. The client computes its local second moment \bm{V}_{k}=(\bm{G}_{k})^{\odot 2} . Baselines. We compare three aggregation strategies to estimate \bm{V}^{\star} given \{V_{k}\}_{k=1}^{K} : 1. Naive Averaging: \bar{\bm{V}}=\frac{1}{K}\sum_{k}\bm{V}_{k} . This is the standard FedOpt approach. 2. Averaging + SVD: We compute the naive average \bar{\bm{V}} and then project it to rank r^{\prime} via SVD. This attempts to denoise the result after aggregation. 3. AJIVE (Ours): We apply AJIVE to the collection \{\bm{V}_{k}\}_{k=1}^{K} to extract the joint spectral component shared across clients, effectively filtering out the drift \bm{L}_{k} before it corrupts the signal. F.2 Results and Analysis We measure the reconstruction error \|\bm{V}_{\text{est}}-\bm{V}^{\star}\|_{F} as a function of the number of participating clients. The results are visualized in Figure 5 . Our observations include: • Failure of Naive Averaging : As shown by the red dashed line, simply averaging local states results in high error. This confirms our theoretical concern: non-linear drift does not cancel out linearly. • Rank Expansion : The element-wise square of a rank- r matrix has an effective rank much larger than r . Consequently, estimators restricted to the original rank (Rank 5, blue lines) underfit the geometry. AJIVE Rank 15 (green circles) matches the theoretical rank of the squared signal, achieving near-perfect recovery as the number of clients increases. • Robustness : AJIVE outperforms post-hoc SVD (solid green line) because it uses the consensus of subspaces across clients to identify the signal, whereas post-hoc SVD cannot distinguish between the signal and the dominant directions of the drift variance. Appendix G Details of Hyper Parameters We implement all methods in PyTorch and use Ray to simulate distributed clients. For GLUE and DomainNet, we use E{=}2 local epochs per round and R{=}50 communication rounds; for Llama-2-7B we use E{=}1 and R{=}20 due to higher per-client cost. Additional details follow the pipeline configs in conf/pipeline/*.yaml . Table 8 : Federated protocol and core training hyperparameters. GLUE/DomainNet use E{=}2 and R{=}50 ; Llama uses E{=}1 and R{=}20 . All runs use bf16. Domain Model LR BS (tr/ev) GA Warmup WD Sched Len GLUE RoBERTa-base ( 36 ) 5\!\times\!10^{-5} 32/32 – 0.06 0.01 cosine – DomainNet ViT-base ( 11 ) 2\!\times\!10^{-4} 128/128 – 0.05 0.1 cosine – MetaMathQA Llama-2-7B ( 55 ) 1\!\times\!10^{-4} 2/2 32 0.03 0.01 cosine 1024 Table 9 : LoRA configuration and target modules. Domain r \alpha Task Modules to save Target modules GLUE 8 16 SEQ_CLS ["classifier"] ["query","value","dense"] DomainNet 16 32 SEQ_CLS ["classifier"] ["query","value","dense"] MetaMathQA 32 64 CAUSAL_LM null ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"] Table 10 : Compute allocation. GPUs per run in our Ray-based client simulation. Domain #GPUs GLUE 0.5 DomainNet 0.5 MetaMathQA (Llama-2-7B) 1 Appendix H Dirichlet Distribution for Modeling Non-IID Data (a) \alpha=10 (b) \alpha=1 (c) \alpha=0.5 Figure 6 : Impact of concentration parameter \alpha on data distribution. We illustrate the class distribution across clients for varying \alpha on MNIST datasets. Smaller values (e.g., \alpha=0.5 ) result in severe class imbalance, where clients possess samples from very few classes. Following common practice in federated learning ( 69 ; 57 ; 30 ) , we model client data heterogeneity using a Dirichlet distribution. Specifically, for a classification task with K classes, the label distribution for each client i is sampled from a Dirichlet distribution: p_{i}\sim\text{Dir}(\alpha\cdot\mathbf{1}_{K}), where \alpha>0 is the concentration parameter and \mathbf{1}_{K} is a K -dimensional vector of ones. A smaller \alpha produces more skewed client label distributions (i.e., stronger non-IID conditions), while a larger \alpha yields more balanced distributions approaching the IID case. After sampling p_{i} , we partition the dataset by allocating examples to client i according to p_{i} . Figure 6 illustrates how the Dirichlet partition leads to distinct label distributions across clients. For MetaMathQA ( 68 ) , we treat “type” as the label. In our experiments, we set \alpha=0.5 to study the impact of severe data heterogeneity. Appendix I More Results of GLUE Table 11 : GLUE Benchmark Results. ✓ denotes IID. ✕ denotes Non-IID, \Delta indicates the difference. Method MNLI Acc% QNLI Acc% RTE Acc% STS-B MSE ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta ✓ ✕ \Delta FedAvg-Full 82.5 81.1 \downarrow 1.4 90.1 89.8 \downarrow 0.3 75.1 68.2 \downarrow 6.9 0.45 0.67 \downarrow .22 FedIT 75.6 69.8 \downarrow 5.8 83.4 78.1 \downarrow 5.3 66.5 58.9 \downarrow 7.6 0.38 0.55 \uparrow .17 FFA-LoRA 82.2 75.8 \downarrow 6.4 89.9 83.4 \downarrow 6.5 74.8 65.3 \downarrow 9.5 0.44 0.56 \uparrow .16 FLoRA 82.7 73.2 \downarrow 9.5 92.0 86.7 \downarrow 5.3 75.1 68.2 \downarrow 6.9 0.39 0.53 \uparrow .14 LoRA-Fair 81.3 80.0 \downarrow 1.3 83.1 81.8 \downarrow 1.3 71.3 63.2 \downarrow 8.1 0.57 0.61 \uparrow .04 FR-LoRA 80.9 72.3 \downarrow 8.6 93.1 90.8 \downarrow 2.3 76.2 70.4 \downarrow 5.8 0.33 0.39 \uparrow .06 FedGaLore - 81.2 77.4 \downarrow 3.8 93.3 88.1 \downarrow 5.2 76.2 68.0 \downarrow 8.2 0.33 0.51 \uparrow .18 FedGaLore 82.7 81.4 \downarrow 1.3 90.3 89.8 \downarrow 0.5 75.2 68.2 \downarrow 7.0 0.45 0.67 \uparrow .22