FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC, Australia DOI: 10.1145/3664647.3681094 ISBN: 979-8-4007-0686-8/24/10 CCS: Theory of computation Distributed algorithms

Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, YunXiang Gong · 2026

Read FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC, Australia DOI: 10.1145/3664647.3681094 ISBN: 979-8-4007-0686-8/24/10 CCS: Theory of computation Distributed algorithms by Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, YunXiang Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05116.txt.

Introduction

Junkang Liu and team offer FedBCGD, a federated block coordinate gradient descent method splitting model settings into blocks, with each client uploading a given block to cut communication overhead. An accelerated variant, FedBCGD+, adds client drift control and stochastic spread cut. The paper claims priority on setting block communication for large-scale deep models. Settling analysis shows communication hardness lower than current methods by a factor of 1/N, where N counts setting blocks, plus much faster settling. Results show superiority over state-of-the-art algorithms.

Catalog record

CREATORJunkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, YunXiang Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE145352
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC, Australia DOI: 10.1145/3664647.3681094 ISBN: 979-8-4007-0686-8/24/10 CCS: Theory of computation Distributed algorithms Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, YunXiang Gong Abstract . Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks, including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor 1 / N 1/N lower than those of existing methods, where N N is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms. The code is available at https://github.com/junkangLiu0/FedBCGD . 1. Introduction Figure 1 . The diagram of the proposed FedBCGD framework, where S\geq N , S and N are the numbers of clients and parameter blocks, respectively. Federated Learning (FL) is an emerging machine learning paradigm, which aims at achieving collaborative model training among multiple parties to preserve data privacy. FL achieves model training by training models locally on client devices and then uploading them to a central server for model aggregation ( 37 ) . Compared to centralized learning in a data center ( 12 ) , the parallel computing clients of FL have private data stored in them and communicate remotely with a central server. The clients are responsible for local training, while the central server in charge of aggregating the models uploaded by each client. Currently, FL has been widely applied in different fields such as mobile intelligence devices, medical health, and financial risk control ( 45 ; 3 ; 6 ) . Our theoretical results show that the communication complexities of our algorithms are a factor 1/N lower than those of existing methods, where N is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. In mainstream frameworks for federated learning, the communication between clients and their server is slow, costly, and unreliable ( 20 ) . In recent years, large models such as BERT and ChatGPT ( 8 ; 4 ) have emerged, leading to an exponential increase in the model size and data volume on FL clients. The upload of these large models further escalates the cost of communication in FL. To significantly lower the communication complexity, this paper proposes a novel method for FL, called Federated Block Coordinate Gradient Descent (FedBCGD) based on block coordinate descent (BCD) methods ( 48 ) . Table 1 . Comparison of the communication complexities and communication overheads of different algorithms in the \mu -strongly convex setting, where \sigma is the variance of stochastic gradients, G is heterogeneity due to client data distribution, S is the number of participating clients, K\!=\!S/N , K is the number of clients involved in each parameter block, N is the number of parameter blocks, and T is the number of local training iterations. The number of floats sent per round by FedAvg is d , and \mathcal{O} describes the worst-case complexity of different algorithms, where \alpha=\frac{1}{1-\lambda} . In non-convex settings, \tau is the second-order heterogeneity (see ( 17 ) ), G is the first-order heterogeneity, F:=f\left(\boldsymbol{x}^{0}\right)-f^{\star} , and f^{\star} is a minimum value of Problem (1) below. Algorithm Strongly convex Communication complexity Non-convex Communication complexity Client sample Stochastic Gradient Floats sent per round FedAvg ( 37 ) \mathcal{O}\Big(\frac{\sigma^{2}+G^{2}}{\mu ST\epsilon}+\frac{\sigma+G}{\mu\sqrt{\epsilon}}+\frac{\beta}{\mu}\log\frac{1}{\epsilon}\Big) \mathcal{O}\Big(\frac{\beta\sigma^{2}}{TS\epsilon^{2}}+\frac{\sqrt{\beta}G+\sqrt{\frac{\beta}{T}}\sigma}{\epsilon^{\frac{3}{2}}}+\frac{F\beta}{\epsilon}\Big) Yes Yes d FedBCGD (ours) \mathcal{O}\Big(\frac{\sigma^{2}+G^{2}}{\mu ST\epsilon}+\frac{\sigma+G}{\alpha\mu N\sqrt{\epsilon}}+\frac{\beta}{\mu N}\log\frac{1}{\epsilon}\Big) \mathcal{O}\Big(\frac{\beta\sigma^{2}}{TS\epsilon^{2}}+\frac{\sqrt{\beta}G+\sqrt{\frac{\beta}{T}}\sigma}{N\epsilon^{\frac{3}{2}}}+\frac{F\beta}{N\epsilon}\Big) Yes Yes d/N SCAFFOLD ( 18 ) \mathcal{O}\Big(\frac{\sigma^{2}}{\mu ST\epsilon}\!+\!\frac{\sigma}{\mu\sqrt{\epsilon}}\!+\!\Big(\frac{M}{S}\!+\!\frac{\beta}{\mu}\Big)\log\frac{1}{\epsilon}\Big) \mathcal{O}\Big(\frac{\beta\sigma^{2}}{TS\epsilon^{2}}+\frac{\sqrt{\frac{\beta}{T}}\sigma}{\epsilon^{\frac{3}{2}}}+\frac{\beta F}{\epsilon}\Big(\frac{M}{S}\Big)^{\frac{2}{3}}\Big) Yes Yes 2d FedLin ( 39 ) \mathcal{O}\Big({\frac{\beta}{\mu}}\log\frac{1}{\epsilon}\Big) —— No No 2d S-Local-GD ( 11 ) \mathcal{O}\Big({\frac{\beta}{\mu}}\log\frac{1}{\epsilon}\Big) —— No No 2d CE-LSGD ( 41 ) —— \mathcal{O}\Big(\frac{GF\tau}{M\epsilon^{\frac{3}{2}}}\Big) Yes Yes 3d BVR-L-SGD ( 40 ) —— \mathcal{O}\Big(\frac{F\tau}{\epsilon}\!+\!\frac{F\beta}{\sqrt{T}\epsilon}\!+\!\frac{\sigma^{2}}{MT\epsilon}\!+\!\left(\frac{\sigma F\beta}{MT\epsilon}\right)^{\frac{3}{2}}\Big) Yes Yes 3d FedBCGD+ (ours) \mathcal{O}\left(\left(\frac{M}{S}+\sqrt{\frac{\beta}{\mu}}\right)\log\frac{1}{\epsilon}\right) \mathcal{O}\big(\frac{\beta F}{\epsilon}\big(\frac{M}{S}\big)^{\frac{2}{3}}\frac{1}{N}^{\frac{1}{3}}\big) Yes Yes 2d/N In FL, the upload speed of the client model is more than a hundred times slower than the download speed, so this paper mainly resolves the issue of upload communication cost. As shown in Figure 1 , we divide the model parameter \boldsymbol{x} into N blocks and \boldsymbol{x}_{s} , i.e., \boldsymbol{x}=\big[\boldsymbol{x}_{(1)}^{\top},\ldots,\boldsymbol{x}_{(N)}^{\top},\boldsymbol{x}_{s}^{\top}\big]^{\top} , where \boldsymbol{x}_{s} denotes the shared parameters in each client (usually the parameters of the last layer classifier, and their number is small but important, ( 35 ) suggests that the deeper the model, the greater the variance of the parameters. In FL, it is often the parameters in the last layer of the classifier that are most important and have a very small number of covariates (0.01% of the overall number in ResNet-18)). Each client is responsible for optimizing one selected parameter block \boldsymbol{x}_{(j)} and shared parameter block \boldsymbol{x}_{s} . After local training for all model parameters, the updated parameter block \boldsymbol{x}_{(j)} and shared parameter block \boldsymbol{x}_{s} are sent to the server, which takes average aggregation of parameters for different parameter blocks to get the complete model. The initial idea is to require each client to perform local updates only on the specified parameter block \boldsymbol{x}_{(j)} and \boldsymbol{x}_{s} while freezing the remaining parameter blocks (called FedBCGD_freezing). After local training, the specified parameter blocks would be uploaded for model aggregation. However, due to a large drift between parameter blocks, such scheme often results in bad convergence in our experiments (see Figure 5 for details). More specifically, only updating certain parameter blocks locally results in a large gap between the updated parameter blocks and other freezing parameter blocks, and it is not possible to establish good connections between parameter blocks during the server-side aggregation process. Therefore, we propose a novel FedBCGD method to address these issues. In the proposed algorithm, we employ stochastic gradient descent to update all parameters instead of parameter freezing during local training, but only transmit two specified parameter blocks ( \boldsymbol{x}_{(j)} and \boldsymbol{x}_{s} ) during the upload process. In addition, to compensate for some missing parameters in block parameter transmission, we add parameter block momentum on the server side. This algorithm design maintains the advantages of low communication costs and has demonstrated a significantly improved convergence speed in our experiments (see Figure 5 for details). Moreover, adding one shared parameter block in each client can significantly improve accuracy performance. However, due to the impact of data heterogeneity, it still leads to inconsistent update directions between parameter blocks, called parameter block drift, resulting in poor performance of the aggregated model. Thus, we also propose an accelerated version (called FedBCGD+)

Read the complete text