Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Shuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto · 2025
Read Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA by Shuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01755.txt.
Description
Shuangyi Chen and five colleagues from the University of Toronto and Ericsson wrote a 2025 paper on fine-tuning large language models across many devices without centralizing the data. Federated learning trains models where the data lives, and LoRA is a popular low-cost adapter method that updates only small added matrices. The challenge is keeping federated LoRA both expressive and robust.
Their framework, RoLoRA, fine-tunes the LoRA adapters by alternating optimization, taking turns with the up and down projection matrices. The authors argue this improves the adapters' expressiveness compared with earlier schemes that produced imperfect updates. A theoretical analysis on a linear model highlights why the alternation matters, and extensive experiments support the advantages. Robust federated fine-tuning matters for privacy-sensitive applications.
Catalog record
| CREATOR | Shuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 185 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01755