FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Chuiyang Meng, Ming Tang, Vincent W.S. Wong · 2026

Read FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment by Chuiyang Meng, Ming Tang, Vincent W.S. Wong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17095.txt.

Description

Chuiyang Meng led a three-author team fixing a subtle flaw in federated fine-tuning of large language models. Low-rank adaptation splits tuning into two small matrices. Federating it creates two headaches. Aggregating the matrices separately introduces error, and aggregating their product forces a re-split that is not unique, so updates drift. The paper proposes FLoRG, a framework built on a single low-rank matrix.

Instead of aggregating matrices, FLoRG aggregates the Gram matrix of inner products, which removes the aggregation error and trims communication. A Procrustes alignment step lines up decompositions across rounds, cutting drift and tightening the convergence bound, which the authors prove in theory. Across several fine-tuning benchmarks, FLoRG beat five strong baselines on downstream accuracy while lowering communication cost.

Catalog record

CREATORChuiyang Meng, Ming Tang, Vincent W.S. Wong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17095

Browse the full library · About the Tower of Babel