Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation
Zikai Zhang Rui Hu Jiahao Xu · 2026
Read Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation by Zikai Zhang Rui Hu Jiahao Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-16936.txt.
Description
Zikai Zhang, Rui Hu, and Jiahao Xu propose Fed-PLoRA, a lightweight mixed-resource joint fine-tuning frame for large language models. Federated learning enables joint fine-tuning across distributed clients using low-rank adaptation, preserving privacy by avoiding raw data sharing. Yet deployments struggle when clients hold mixed resources and thus adopt varied low-rank adaptation ranks, causing substantial initialization and aggregation noise that undermines performance.
Fed-PLoRA brings in Parallel One-Rank Adaptation, a new low-rank adaptation variant replacing the classic multi-rank module with multiple parallel one-rank modules. A Select-N-Fold strategy folds untrained modules into pretrained weights before local training. A unified study of initialization and aggregation noise shows how Fed-PLoRA addresses limits of state-of-the-art methods. Extensive tests on varied fine-tuning tasks show Fed-PLoRA consistently beats existing methods, giving joint deployments a practical route to mixed clients.
Catalog record
| CREATOR | Zikai Zhang Rui Hu Jiahao Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-16936