Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation

Zikai Zhang Rui Hu Jiahao Xu · 2026

Read Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation by Zikai Zhang Rui Hu Jiahao Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-16936.txt.

Description

Zikai Zhang, Rui Hu, and Jiahao Xu propose Fed-PLoRA, a lightweight mixed-resource joint fine-tuning frame for large language models. Federated learning enables joint fine-tuning across distributed clients using low-rank adaptation, preserving privacy by avoiding raw data sharing. Yet deployments struggle when clients hold mixed resources and thus adopt varied low-rank adaptation ranks, causing substantial initialization and aggregation noise that undermines performance.

Fed-PLoRA brings in Parallel One-Rank Adaptation, a new low-rank adaptation variant replacing the classic multi-rank module with multiple parallel one-rank modules. A Select-N-Fold strategy folds untrained modules into pretrained weights before local training. A unified study of initialization and aggregation noise shows how Fed-PLoRA addresses limits of state-of-the-art methods. Extensive tests on varied fine-tuning tasks show Fed-PLoRA consistently beats existing methods, giving joint deployments a practical route to mixed clients.

Catalog record

CREATORZikai Zhang Rui Hu Jiahao Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-16936

Browse the full library · About the Tower of Babel