FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Models

Guohao Yang ∗ \ast , Tongle Wu ⋆ \star , Yuanxiong Guo ∗ \ast , Ying Sun ⋆ \star , and Yanmin Gong ∗ \ast · 2026

Read FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Models by Guohao Yang ∗ \ast , Tongle Wu ⋆ \star , Yuanxiong Guo ∗ \ast , Ying Sun ⋆ \star , and Yanmin Gong ∗ \ast (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03019.txt.

Description

Fine-tuning large models across many private devices is made cheaper by Guohao Yang and 4 co-authors in a 2026 paper. Federated learning trains collaboratively without centralizing data, keeping privacy. Full fine-tuning, though, costs too much communication and memory for resource-constrained clients, while lighter setting-efficient methods often lose results compared with the full version.

Their method, FedKRSO, updates models inside shared random low-part subspaces made by the server. Clients send only compact accumulators along those subspaces, not full settings. The server aggregates and redistributes the global model efficiently. Convergence is analyzed rigorously for general federated settings. Tests on the GLUE benchmark across scenarios showed strong results with low overhead, approaching full federated fine-tuning quality.

Catalog record

CREATORGuohao Yang ∗ \ast , Tongle Wu ⋆ \star , Yuanxiong Guo ∗ \ast , Ying Sun ⋆ \star , and Yanmin Gong ∗ \ast
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03019

Browse the full library · About the Tower of Babel