Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA

Shuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto · 2025

Read Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA by Shuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01755.txt.

Description

Shuangyi Chen and five colleagues from the University of Toronto and Ericsson wrote a 2025 paper on fine-tuning large language models across many devices without centralizing the data. Federated learning trains models where the data lives, and LoRA is a popular low-cost adapter method that updates only small added matrices. The challenge is keeping federated LoRA both expressive and robust.

Their framework, RoLoRA, fine-tunes the LoRA adapters by alternating optimization, taking turns with the up and down projection matrices. The authors argue this improves the adapters' expressiveness compared with earlier schemes that produced imperfect updates. A theoretical analysis on a linear model highlights why the alternation matters, and extensive experiments support the advantages. Robust federated fine-tuning matters for privacy-sensitive applications.

Catalog record

CREATORShuangyi ChenUniversity of Toronto, Yuanxin Guo University of Toronto, Yue JuEricsson-GAIA Montréal, Hardik DalalEricsson-GAIA Montréal, Zhongwen ZhuEricsson-GAIA Montréal, Ashish KhistiUniversity of Toronto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE185 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-01755

Browse the full library · About the Tower of Babel