A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning

Tongtian Zhu, Tianyu Zhang, Mingze Wang, Zhanpeng Zhou, Can Wang1 Zhejiang University 2 Mila - Quebec AI Institute 3 Université de Montréal 4 Peking University 5 Shanghai Jiao Tong University {raiden, wcang}@zju.edu.cn, [email protected] [email protected] [email protected] · 2025

Read A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning by Tongtian Zhu, Tianyu Zhang, Mingze Wang, Zhanpeng Zhou, Can Wang1 Zhejiang University 2 Mila - Quebec AI Institute 3 Université de Montréal 4 Peking University 5 Shanghai Jiao Tong University {raiden, wcang}@zju.edu.cn, [email protected] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06542.txt.

Introduction

Tongtian Zhu, Can Wang and their co-authors show that a single global merging step can restore server-level performance in decentralized learning in 2025. Decentralized training scales well, but limited peer-to-peer talk often drags its accuracy down.

Experiments found that saving communication for the later stages of training helped global test scores a lot. A single full merging at the final step gave large gains even when local data differed sharply. Their theory is the first to show the merged model can match the convergence rate of parallel server training. Part of the gap between local models, once seen as noise, is actually constructive.

Catalog record

CREATORTongtian Zhu, Tianyu Zhang, Mingze Wang, Zhanpeng Zhou, Can Wang1 Zhejiang University 2 Mila - Quebec AI Institute 3 Université de Montréal 4 Peking University 5 Shanghai Jiao Tong University {raiden, wcang}@zju.edu.cn, [email protected] [email protected] [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06542

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning Tongtian Zhu, Tianyu Zhang, Mingze Wang, Zhanpeng Zhou, Can Wang1 Zhejiang University 2 Mila - Quebec AI Institute 3 Université de Montréal 4 Peking University 5 Shanghai Jiao Tong University {raiden, wcang}@zju.edu.cn, [email protected] [email protected] [email protected] Abstract Decentralized learning provides a scalable alternative to traditional parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. Our empirical results show that concentrating communication budgets in the later stages of decentralized training markedly improves global generalization. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, is sufficient to match the performance of server-based training. We further show that low communication in decentralized learning preserves the mergeability of local models throughout training. Our theoretical contributions, which explains these phenomena, are first to establish that the globally merged model of decentralized SGD can converge faster than centralized mini-batch SGD. Technically, we novelly reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components that accelerate convergence. This work challenges the common belief that decentralized learning generalizes poorly under data heterogeneity and limited communication, while offering new insights into model merging and neural network loss landscapes. The code will be made publicly available soon. 1 Introduction Decentralized learning offers a promising approach to crowdsource computational workloads across geographically distributed compute [ 100 , 6 , 39 ] . A defining characteristic of this setting is the reliance on peer-to-peer communication during training, involving the peer-level exchange of model parameters or gradients during training. However, such communication is often constrained in practice due to limited bandwidth between geographically distant nodes, making it a scarce resource. These constraints can significantly degrade the performance of decentralized learning, both theoretically and empirically [ 53 , 42 , 86 ] . As a result, efficiently allocating limited communication resources becomes a fundamental challenge in decentralized learning, especially in heterogeneous environments where local data distributions vary across nodes [ 58 ] . To date, most efforts addressing this challenge have focused on optimizing communication allocation at the spatial level , particularly through the design of communication graphs [ 99 , 51 , 82 , 40 ] . In contrast, few works have attempted to study how communication should be allocated over temporal levels, i.e., when and how frequently agents synchronize parameters or gradients with others. Exploring the communication allocation problem at the temporal levels is of significant importance to improve the efficiency and generalization of decentralized learning. Question : How to allocate communication budget in decentralized learning over temporal levels? To answer this question, we design a series of experiments that allocate communication budgets across different time windows during training (see Fig. 3 ). Specifically, we divide the training process into consecutive windows, each consisting of a fixed number of communication rounds. We assign higher communication budgets to selected windows using global synchronization via AllReduce [ 76 ] , while keeping communication low otherwise by infrequent synchronization with random peer agents. 1 1 1 Agents here refer to participants or nodes in the decentralized learning system. This setup enables us to systematically examine how the temporal communication allocations influence final global performance under constrained budgets. We observe that allocating higher communication budgets toward the later stages of training consistently leads to improved final global test performance. More surprisingly, we find that even a single round of fully-connected communication 2 2 2 Fully-connected communication refers to global synchronization via AllReduce . In this paper, fully-connected communication is realized through parameter averaging over the models on all agents, namely global merging . at the final step, implemented as a global merging, can recover the server-based training 3 3 3 “Server-based training” refers to training schemes coordinated by central servers, e.g., federated learning. See Fig. 2 for a comparative illustration of different training schemes. performance (see Fig. 1(a) and Fig. 1(b) ). See Fig. 2 for a comparative illustration of different training schemes. Surprising Phenomenon : A single global merging of decentralized models, even under severely constrained communication and high data heterogeneity, can recover server-based training performance. Further results show that, throughout training, the average of local models in decentralized learning consistently achieves higher global test accuracy than individual models (see Fig. 1(c) and Fig. 3(c) ), a property we refer to as mergeability (see Definition 1 ). While extensive studies suggest that merging independently trained models typically leads to an increased loss [ 21 , 1 ] , we show that local models in decentralized learning remain mergeable and can approach and even surpass the performance of server-based training after merging. Our findings challenge the conventional belief that decentralized learning fails to generalize under high data heterogeneity and limited communication, and offer new insights into model merging techniques [ 91 , 72 ] and loss landscape of neural networks [ 21 , 18 ] . ((a)) CLIP ViT-B/32 ((b)) ResNet-18 (w/o pretraining) ((c)) ResNet-18 (w/o pretraining) Figure 1 : (a, b) : Comparisons of global test accuracy (see Definition 2 ) in training of CLIP ViT-B/32 (a) and ResNet-18 (b) on the Tiny ImageNet dataset, distributed across 16 agents with high heterogeneity (Dirichlet \alpha = 0.1; see details in Section C.1 ). Decentralized training here involves each agent syncing with a random peer per round. A single global model merging is performed at the final round. (c) : Loss landscape of decentralized training (w/o permutation) before global merging, under the same setup as the orange curve in (b); {\{\theta_{k}\}}_{k=1}^{16} denotes the local models. Subsequently, we delve deeper into the mechanism behind the mergeability of local models in decentralized learning. We make an initial attempt by analyzing the convergence of decentralized SGD (DSGD) in a finer-grained manner. We theoretically establish that the globally merged model of DSGD can achieve faster convergence than centralized mini-batch SGD [ 15 , 49 ] . 4 4 4 “Centralized mini-batch SGD” refers to parallel SGD training, with synchronization realized by a central server. Our key technical novelty lies in a novel treatment of consensus violation, which characterizes the discrepancy among local models. In contrast to prior work that considers such violations as purely detrimental noise, we partially incorporate them as constructive components to accelerate convergence. Figure 2 : A comparative illustration of server-based, decentralized, and local training. Our Contributions. In summary, our work focuses on the temporal communication allocation in decentralized learning, which has not been carefully studied yet. We show that a single global merging at the end of training is sufficient for decentralized learning to match the global generalization performance of server-based learning. We summarize our main contributions as follows. • Empirical Observations . (1) : We highlight the critical role of allocating communication towards the later stages of decentralized training. Remarkably, a single global merging at the final step can recover server-based training performance, even under severe communication constraints and data heterogeneity (see Fig. 1(a) , Fig. 1(b) and additional results in Section C.2 ). (2) : Low communication throughout training preserves the mergeability of local models throughout

Read the complete text