LOSCAR -SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
Yassine Maziane, Ammar Mahran, Artavazd Maranjyan, Peter Richtárik · 2026
Read LOSCAR -SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging by Yassine Maziane, Ammar Mahran, Artavazd Maranjyan, Peter Richtárik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20866.txt.
Introduction
Yassine Maziane and three colleagues propose LOSCAR-SGD, local SGD with communication-computation overlap and delay-corrected sparse model averaging. Communication is a big bottleneck in distributed learning, especially at large scale and in federated settings with slow links. Three standard cost cutters are communication compression, local learning, and communication-computation overlap. Methods combining all three work in practice but lack theory, and LOSCAR-SGD studies a heterogeneous-compute setting where workers take different numbers of local steps.
Catalog record
| CREATOR | Yassine Maziane, Ammar Mahran, Artavazd Maranjyan, Peter Richtárik |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110660 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LOSCAR -SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging Yassine Maziane, Ammar Mahran, Artavazd Maranjyan, Peter Richtárik Abstract Communication is a major bottleneck in distributed learning, especially in large-scale settings and in federated learning environments with slow links. Three standard ways to reduce this cost are communication compression, local training, and communication-computation overlap. Methods that combine these ingredients are used in practice and have been found to be effective for large-scale training, but there is little theory for methods that combine all three. We study a heterogeneous-compute setting in which different workers may take different numbers of local steps, and we propose LOSCAR - SGD , a Local SGD method that communicates only a sparse subset of model coordinates and continues optimizing while communication is in flight. A key ingredient is a delay-corrected merge rule that incorporates delayed synchronized information without discarding the progress made during the overlap phase. We give convergence guarantees for smooth non-convex objectives and show how sparsity, overlap, and worker heterogeneity affect the rate. To the best of our knowledge, this is the first theory for this combination of ingredients. Experiments further show that communication-computation overlap reduces training time and that the delay-corrected merge outperforms naive overwriting. 1 Introduction Distributed learning enables the training of large models across many workers. The most common setup is data parallelism , where the data is split across workers, each worker keeps a copy of the model, and the workers communicate to train one shared model ( 14 ; 22 ; 51 ) . A simple baseline in this setting is synchronized Minibatch SGD : at every iteration, each worker computes one stochastic gradient, the gradients are averaged, and one global update is taken. This is simple, but it requires communication at every iteration. As models become larger and the number of workers grows, communication can become the main factor limiting training speed. Federated learning is an especially clear example of this issue, since communication often happens over the internet and is much slower than local computation ( 34 ; 17 ; 45 ) . There are several common ways to reduce this communication cost. One is local training : instead of synchronizing after every stochastic-gradient step, workers perform several local SGD steps before they communicate. Because the local steps are simple SGD steps, this approach is often called Local SGD ( 52 ) , and in the federated learning literature it became popular through FedAvg ( 34 ) . Another idea is communication-computation overlap . In standard synchronized methods such as Minibatch SGD and plain FedAvg -style local training, workers may sit idle while messages are being sent, aggregated, and returned. With overlap, workers keep optimizing locally while communication is in flight, which can hide part of the communication delay and reduce training time ( 47 ; 41 ; 18 ) . A third idea is to reduce the amount of communicated information through compression , for example via quantization or sparsification ( 3 ; 48 ) . In local training, workers often synchronize model weights by averaging them; with sparsification, only a subset of model coordinates is synchronized ( 12 ; 4 ) . These techniques are each well studied in both theory and practice. There is also practical evidence that combining them can work well ( 9 ; 2 ) . However, to the best of our knowledge, there is no theory that covers the full combination of local training, sparse synchronization, and communication-computation overlap. The problem is further complicated when workers have different computation speeds, as in heterogeneous clusters or federated settings, because different workers may perform different amounts of local work in the same amount of time. A natural way to handle this is to let different workers take different numbers of local steps so that they remain roughly aligned in time ( 23 ; 30 ; 31 ) . In this work, we study the data-homogeneous version of this setting and propose LOSCAR - SGD , a Local SGD -type method with infrequent sparse model averaging, communication-computation overlap, and worker-specific local-step counts. We also study a delay-corrected merge rule that preserves the progress made during the overlap phase instead of discarding it through a naive averaging overwrite. Table 1 summarizes representative methods from the literature that incorporate at least one of the ingredients discussed above and have accompanying theoretical analysis. Methods Local steps Compression Comm./comp. overlap Worker-specific local-step counts Minibatch SGD ✘ ✘ ✘ ✘ Local SGD , FedAvg ( 52 ; 34 ) ✔ ✘ ✘ ✘ SPARTA , LoCoDL ( 4 ; 6 ) ✔ ✔ ✘ ✘ Overlap-Local-SGD , CO2 ( 47 ; 41 ) ✔ ✘ ✔ ✘ FedProx / GradSkip ( 23 ; 31 ) ✔ ✘ ✘ ✔ LOSCAR - SGD ( Algorithm 1 ) ✔ ✔ ✔ ✔ Table 1: Comparison of representative distributed learning methods with theoretical analysis related to our setting. The columns indicate whether a method explicitly incorporates local steps, compression, communication-computation overlap, and worker-specific local-step counts. A checkmark indicates the presence of the corresponding feature. To the best of our knowledge, our method is the first analyzed method that combines all four features. 1.1 Contributions • We introduce a general Local SGD framework ( Algorithm 1 ) for data-homogeneous distributed optimization that combines four ingredients in one model: local training, sparse model averaging, communication-computation overlap, and worker-specific local-step counts. • We propose a delay-corrected merge rule for sparse synchronization. When delayed synchronized information arrives, the worker should not discard the local progress it made during communication. Instead of simply overwriting coordinates with a delayed average, the rule keeps the overlap progress and corrects only the synchronization disagreement. • In Section 4 , we prove convergence guarantees for smooth non-convex objectives. The bounds show how sparsity, overlap, and heterogeneity affect convergence. They also show that the extra error caused by overlap-induced staleness appears as a controlled higher-order term. • Experiments in Section 5 show that communication-computation overlap reduces training time, and that the delay-corrected merge is better than naive overwriting. 1.2 Related work Local training Local training reduces communication by performing multiple stochastic-gradient steps between synchronizations. It was popularized in federated learning by FedAvg ( 34 ) , building on earlier distributed-training ideas ( 52 ; 38 ; 37 ) . Federated-learning analyses in the heterogeneous-data regime include ( 24 ; 19 ; 36 ; 10 ) , while for our setting the most relevant theory is the homogeneous-data Local SGD line ( 50 ; 15 ; 46 ; 49 ) ; for broader background, see 27 . Compression and sparsification Another standard way to reduce communication is to transmit compressed information. Classical work studies gradient compression, sparsification, and quantization ( 3 ; 48 ; 25 ; 5 ) , but for methods based on periodic model averaging the closer references are those that sparsify the communicated model itself ( 4 ; 11 ; 9 ; 6 ; 7 ) . Communication-computation overlap Communication-computation overlap is one of the main ingredients of our method. Instead of leaving workers idle while messages are sent, aggregated, and returned, overlap lets them continue taking useful local updates while communication is in flight, which can