Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment

Hongyi Peng, Han Yu, Xiaoxiao Li, Qiang Yang · 2026

Read Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment by Hongyi Peng, Han Yu, Xiaoxiao Li, Qiang Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01746.txt.

Description

Hongyi Peng and colleagues fix federated LoRA tuning, which falls short of full tuning when client data differ. Analysis traces the gap to two mismatches. Clients optimize in a low rank subspace while aggregation runs in full space, and unsynchronized optimizer states amplify drift. Their FedGaLore pairs client GaLore style gradient subspace optimization with server synchronization of projected second moment states that stays robust to drift through spectral shared signal extraction. Across language, vision and generation benchmarks it improves robustness and accuracy over federated LoRA baselines under non-IID settings.

Catalog record

CREATORHongyi Peng, Han Yu, Xiaoxiao Li, Qiang Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91343
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01746

Browse the full library · About the Tower of Babel