ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
Yujia Wang, Yuanpu Cao, Jinghui Chen · 2025
Read ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models by Yujia Wang, Yuanpu Cao, Jinghui Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19959.txt.
Introduction
Yujia Wang, Yuanpu Cao, and Jinghui Chen present ParaBlock, a federated learning method for large language models. In block coordinate federated training, each client trains only part of the model, which saves local work. Even one block can hold so many parameters, though, that sending it dominates the time, above all for weak clients.
ParaBlock runs communication and computation in two parallel threads, so sending and training overlap instead of taking turns. The authors prove it keeps the same convergence rate as standard federated block coordinate methods. Fine-tuning tests on instruction following and math reasoning show strong performance with clearly better communication efficiency. The method suits federated settings where bandwidth, not computing, is the bottleneck.
Catalog record
| CREATOR | Yujia Wang, Yuanpu Cao, Jinghui Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19959
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models Yujia Wang, Yuanpu Cao, Jinghui Chen Abstract Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency. 1 Introduction Federated learning (FL) ( mcmahan2017communication ) is known as a promising privacy-preserving solution, as it keeps data on local clients without exposing sensitive information. A typical FL framework consists of a central server and multiple local clients operating in a single-threaded design. In this setup, clients repeatedly train local models, transmit the updates to the server for aggregation, and wait for the updated model from the server to continue training in the next round. While standard FedAvg type frameworks ( mcmahan2017communication ; karimireddy2020scaffold ; li2020fedprox ; wang2023lightweight ; wang2024tackling ) have demonstrated success across various applications ( koutsoubis2024privacy ; beutel2020flower ) , federated block coordinate descent schemes ( liu2019communication ; wu2021federated ; liu2024fedbcgd ; wang2024save ) have recently gained increasing attention. Such schemes integrate local block coordinate descent (BCD) update strategies into FL, allowing clients to train only a subset of the model (commonly a block) locally instead of the entire model. Although federated BCD still operates in a single-thread manner, where clients need to wait for the model transmission to complete before the next computation step, such communication time is often negligible when models are smaller and the block size is limited. However, with the recent trend of billion-scale large language models (LLMs), such as GPT-3 ( brown2020language ) , Llama 3 ( llama3modelcard ) and Gemma 2 ( team2024gemma ) ), even a single layer can encompass tens to hundreds of millions of parameters. This massive scale significantly increases the communication time between the server and clients. While such communication time was considered negligible in traditional federated block coordinate methods, it has now become a notable factor when fine-tuning LLMs on edge clients. This raises a critical bottleneck for fine-tuning LLMs with federated block coordinate descent methods: for clients with limited network bandwidth or those requiring long-distance transmission, the communication latency significantly impacts the overall efficiency of FL deployment. This inefficiency of such a single-thread approach hinders the scalability of federated block coordinate descent methods for LLMs, underscoring the need for solutions to reduce communication latency for clients. To address the aforementioned challenge, we propose ParaBlock, a federated communication-computation Para llel Block coordinate descent method that provides a simple yet effective way to improve the communication efficiency of BCD updates for fine-tuning large-scale models on resource-constrained clients. The key contributions of this paper are summarized as follows: • We propose ParaBlock, a novel method designed to address communication latency during the fine-tuning of LLMs using federated block coordinate descent. ParaBlock replaces the traditional single-threaded design with two parallel threads for communication and computation, effectively reducing communication delays and improving efficiency. • We rigorously show that the proposed ParaBlock achieves a convergence rate of \mathcal{O}(1/\sqrt{T}) for non-convex objectives. This rate is consistent with that of standard federated block coordinate descent methods, ensuring that the improvements of ParaBlock in communication efficiency do not compromise its convergence performance. • We conduct extensive experiments across various models and tasks to empirically validate the effectiveness of ParaBlock. Specifically, we evaluate the downstream performance of ParaBlock on general instruction following and mathematical reasoning tasks. Our results demonstrate that ParaBlock significantly reduces wall-clock runtime while maintaining performance on par with standard federated block coordinate descent baselines, highlighting its ability to enhance efficiency without compromising utility. 2 Related Works BCD for FL Block Coordinate Descent (BCD) has been extensively studied for optimization problems ( tseng2001convergence ; nesterov2012efficiency ; beck2013convergence ) , particularly due to its efficiency in solving large-scale optimization tasks. Recent advancements in BCD variants, such as those proposed by luo2024badam ; pan2024lisa , further highlight its adaptability and effectiveness, especially in the context of large language models. Federated BCD schemes ( liu2019communication ; liu2024fedbcgd ; wu2021federated ) enhances communication efficiency by assigning each client the responsibility for a partition of the model in a collaborative training framework. FedCyBGD ( wang2024save ) focuses on fine-tuning large language models by cyclically elects one client at a time to train its assigned block of the model, enabling efficient fine-tuning with limited computational resources. Communication-efficiency for FL Various methods have been proposed to improve communication efficiency in federated learning, focusing on techniques such as quantization and update compression ( reisizadeh2020fedpaq ; haddadpour2019local ; jin2020stochastic ; jhunjhunwala2021adaptive ; wang2022communication ; li2024fedbat ) , model pruning ( li2021fedmask ; jiang2022model ; isik2022sparse ) , and model distillation ( wu2022communication ) . Several existing works have demonstrated to enhance communication efficiency specifically in the context of federated fine-tuning of LLMs. For example, many applications of PEFT in FL help reduce communication costs ( liu2019communication ; liu2024fedbcgd ) . Additionally, FedMKT ( fan2024fedmkt ) leverages a mutual knowledge transfer framework, where clients and the server exchange knowledge datasets instead of model updates. Similarly, CG-FedLLM ( wu2024cg ) introduces an encoder on clients to compress gradient features, with a corresponding decoder on the server to reconstruct the transmitted features. FL for LLMs In the era of LLMs, FL has evolved to develop more complex fine-tuning and deployment of large language models ( zhang2024towards ; ye2024openfedllm ) . Several existing approaches explore the use of Parameter Efficient Fine-Tuning (PEFT) techniques for federated fine-tuning of LLMs. These include leveraging low-rank adapters (LoRA) in methods such as FedIT ( zhang2024towards ) , the OpenFedLLM framework ( ye2024openfedllm ) , HetLoRA ( cho2024heterogeneous ) , SLoRA ( babakniya2023slora ) , FLoRA ( wang2024flora ) , FlexLoRA ( bai2024federated ) as well as employing adapters and bias-tuning in FedPEFT ( sun2022conquering ) . In addition to the widely adopted PEFT methods in FL, recent work, such as FedCyBGD ( wang2024save ) , also highlights the potential of BCD in improving federated fine-tuning performance while managing resource limitations. 3 Preliminaries and Motivations We start with a standard federated learning objective with N total workers: \displaystyle\min_{\bm{\theta}\in\mathbb{R}^{d}}f(\bm{\theta}):=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\bm{\theta})=\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}_{\xi_{i}\sim\mathcal{D}_{i}}[f_{i}(\bm{\theta};\xi_{i})], (1) where \bm{\theta}\in\mathbb{R}^{d} indicates the model parameter with d dimensions, f_{i}(\bm{\theta}) represents the loss function associated with client i , and \mathcal{D}_{i} denotes the local data distribution for client i . FedAvg ( mcmahan2017communication ) is widely used for solving ( 1 ). In t -th global round, each participating client i performs local training using standard SGD optimizers. The server periodically collects and aggregates the local models or updates to obtain a new global model. BCD and