Subspace Optimization for Efficient Federated Learning under Heterogeneous Data

Shuchen Zhu, Zhengyang Huang, Yuqi Xu, Peijin Li · 2026

Read Subspace Optimization for Efficient Federated Learning under Heterogeneous Data by Shuchen Zhu, Zhengyang Huang, Yuqi Xu, Peijin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25467.txt.

Introduction

Zhu and three colleagues propose SSF, a subspace method for federated learning. It does heterogeneity-corrected work in a low-dimensional subspace using only projected quantities, with backfill updates that keep full control information. Federated learning runs large models where links, memory, and compute are scarce. Non-IID client data induce drift that hurts local training. Old fixes such as SCAFFOLD correct heterogeneity, but add link and memory costs. Under assumptions, SSF attains a rate of order 1 over T plus 1 over square root of NKT. Tests show good trade-offs.

Catalog record

CREATORShuchen Zhu, Zhengyang Huang, Yuqi Xu, Peijin Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101250
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Subspace Optimization for Efficient Federated Learning under Heterogeneous Data Shuchen Zhu, Zhengyang Huang, Yuqi Xu, Peijin Li Abstract Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degrades the stability and performance of local training. Existing remedies such as SCAFFOLD introduce heterogeneity-correction mechanisms to address this challenge, but they incur substantial extra communication and memory overhead. This paper proposes a subspace optimization method for federated learning (SSF), which performs heterogeneity-corrected optimization in a low-dimensional subspace using only projected quantities, while preserving full-dimensional control information through a backfill-style update that retains residual components whenever the active subspace changes. Under standard smoothness and bounded-variance assumptions, SSF attains a non-asymptotic rate of order 𝒪 ~ ​ ( 1 / T + 1 / N ​ K ​ T ) \widetilde{\mathcal{O}}(1/T+1/\sqrt{NKT}) . Experiments show favorable accuracy–efficiency trade-offs under heterogeneous data. 1 Introduction Federated learning (FL) has emerged as a central paradigm for privacy-preserving distributed training, where a central server coordinates optimization of a shared model across many clients that keep their data local. The standard Federated Averaging (FedAvg) algorithm amortizes communication by allowing multiple local updates between synchronization rounds, and has become a widely adopted baseline due to its simplicity and empirical success in practical deployments [ 16 , 11 , 22 , 17 ] . However, modern FL applications increasingly involve models with hundreds of millions or even billions of parameters and large-scale datasets deployed on resource-constrained edge devices [ 32 , 18 ] . In this large-model regime, federated optimization faces a triple challenge in terms of computation, memory, and communication: each local training step can be computationally heavy, storing full model states stresses limited device memory, and transmitting full-dimensional updates over constrained bandwidth becomes prohibitive. These challenges are further exacerbated in realistic heterogeneous environments where client data distributions differ substantially from each other and from the global distribution. In such non-IID settings, the local objective functions F_{i} become misaligned, inducing client drift in local updates and degrading the stability and convergence of FedAvg and its variants [ 12 , 27 , 26 ] . Heterogeneity-correction methods such as SCAFFOLD address this issue by introducing control variates on the server and clients to correct local gradients, leading to convergence guarantees that resemble centralized training even under strong heterogeneity and partial participation [ 9 ] . While highly effective as robustness baselines in small- to medium-scale settings, these methods are especially costly in the large-model regime: they require storing full-dimensional auxiliary states at each client, communicating additional full-dimensional control variates between the server and clients, and computing full-dimensional gradients at every local step. As a result, full-dimensional heterogeneity-correction methods are at odds with the triple-efficiency requirements of large-scale FL on memory- and bandwidth-limited devices. To alleviate these resource constraints, several complementary lines of work have been explored in federated and distributed optimization. A first line focuses on reducing communication cost via gradient or model compression. Quantization and sparsification techniques such as QSGD and sparse SGD compress updates before transmission, and error-feedback mechanisms compensate for the bias introduced by compression [ 1 , 13 , 25 , 21 , 23 , 24 , 5 , 6 , 7 , 3 ] . These methods can substantially lower the number of bits communicated per round and are widely used in practice. However, they typically operate on full-dimensional gradients or model differences, so the communication cost still scales with the ambient dimension d of the model parameters, and the reduction is mainly in constant factors rather than dimensionality. More critically for large models, error-feedback schemes maintain full-dimensional error buffers to accumulate discarded information, which do not reduce—and often increase —the memory footprint on each client. When combined with full-dimensional control variates or other auxiliary states, compression and error-feedback can therefore aggravate memory pressure even as they reduce raw bandwidth usage. A second line of work leverages subspace and low-rank parameterizations to obtain more holistic efficiency gains. In centralized training, methods such as GaLore project gradients onto low-rank subspaces to reduce the cost of backpropagation while preserving performance [ 33 ] . In federated learning, subspace and low-rank approaches—such as those using low-dimensional adapters, partial model training, or low-rank drift variables in subspace-based federated learning methods [ 31 , 14 ] —constrain optimization to a low-dimensional subspace or factorization. This design yields simultaneous reductions in computation, communication, and memory: local updates are computed in an r -dimensional space with r\ll d , communicated quantities are low-dimensional, and only subspace parameters or low-rank factors need to be stored. However, in strongly heterogeneous FL, these methods often lack robust mechanisms to control client drift. Many subspace FL algorithms do not integrate heterogeneity correction, and naive attempts to introduce low-rank drift compensation variables face serious stability issues: when the underlying subspace frequently changes or rotates, the low-rank compensation variables must be repeatedly re-aligned, making it difficult to maintain effective updates and leading to degraded correction quality and convergence. Consequently, while subspace and low-rank methods excel in efficiency, their convergence robustness under heterogeneity remains limited. More broadly, existing approaches still exhibit a fundamental tension between robustness and efficiency along the three core resource dimensions. Full-dimensional correction and gradient-tracking methods maintain strong control of client drift but incur significant computation, communication, and memory overhead by operating in \mathbb{R}^{d} [ 9 , 10 , 15 ] . Compression with error feedback can mitigate communication costs, yet continues to rely on full-dimensional auxiliary buffers, leaving the memory footprint large or even larger [ 21 , 1 , 13 , 25 , 3 , 23 , 7 ] . Subspace and low-rank FL methods reduce computation, communication, and memory by working in low-dimensional parameterizations [ 33 , 31 , 14 ] , but they do not implement heterogeneity-correction in a principled way. Naive combinations that keep static full-dimensional control variates while training in evolving subspaces either retain the full-dimensional auxiliary states—negating the subspace efficiency benefits—or discard accumulated correction information whenever the subspace changes, thereby undermining the effect of heterogeneity-correction. This state of affairs highlights a key gap: there is currently no federated optimization algorithm that performs native heterogeneity correction in a low-dimensional subspace, so as to (i) preserve robustness to data heterogeneity, while (ii) simultaneously achieving subspace-level efficiency in computation, communication, and memory. Our goal in this work is to fill this gap. Our contributions are as follows. • We propose SSF , a federated optimization algorithm that combines heterogeneity correction with subspace training to address the triple challenge of computation, memory, and communication in large-model FL. SSF constrains all optimization steps to a low-dimensional subspace defined by a projection matrix or low-rank parameterization: local updates and all communicated quantities live in this r -dimensional space, with r\ll d , while the underlying goal remains to optimize the original high-dimensional model parameters for downstream tasks. Crucially, SSF maintains server and client control variates natively in the optimization subspace and uses their difference to correct projected gradients during local updates, mirroring the SCAFFOLD correction mechanism without ever introducing full-dimensional auxiliary states. When the subspace basis is updated or rotated, SSF employs a translation mechanism that consistently maps the accumulated control variates into

Read the complete text