Eager Updates For Overlapped Communication and Computation in DiLoCo

Satyen Kale, Arthur Douillard, Yanislav Donchev · 2025

Read Eager Updates For Overlapped Communication and Computation in DiLoCo by Satyen Kale, Arthur Douillard, Yanislav Donchev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12996.txt.

Introduction

Satyen Kale, Arthur Douillard, and Yanislav Donchev speed up giant model training in a 2025 paper. Training across data centers works in rounds: each site trains alone on its own data for a while, then all sites sync their updates. That sync is cheap next to older methods, yet across distant centers it still forces costly pauses.

Their fix overlaps the two phases instead of blocking on sync. Updates are applied eagerly while messages are still in flight, hiding the wait inside useful work. The paper works out when such overlap stays safe for learning. Data centers could then train huge models with far less idle time.

Catalog record

CREATORSatyen Kale, Arthur Douillard, Yanislav Donchev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12996

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eager Updates For Overlapped Communication and Computation in DiLoCo Satyen Kale, Arthur Douillard, Yanislav Donchev Abstract Distributed optimization methods such as DiLoCo have been shown to be effective in training very large models across multiple distributed workers, such as datacenters. These methods split updates into two parts: an inner optimization phase, where the workers independently execute multiple optimization steps on their own local data, and an outer optimization step, where the inner updates are synchronized. While such approaches require orders of magnitude less communication than standard data-parallel training, in settings where the workers are datacenters, even the limited communication requirements of these approaches can still cause significant slow downs due to the blocking necessary at each outer optimization step. In this paper, we investigate techniques to mitigate this issue by overlapping communication with computation in a manner that allows the outer optimization step to fully overlap with the inner optimization phase. We show that a particular variant, dubbed eager updates, provides competitive performance with standard DiLoCo in settings with low bandwidth between workers. \reportnumber keywords eager updates, distributed learning, large-scale 1 Introduction As language models and data sets get ever larger, it has become increasingly important to resort to distributed training approaches to effectively handle the larger scales. A particularly effective technique, DiLoCo, was proposed by Douillard et al. (2024) . DiLoCo leverages techniques from Federated Learning ( McMahan et al., 2017 ) and uses a particular instantiation similar to the FedOpt algorithm ( Reddi et al., 2021 ) . Specifically, training is split into inner and outer optimization phases. In each inner optimization phase, all workers independently execute an optimizer (typically, AdamW) on their local data starting from the current values of the global parameters. Then, all workers communicate their updates to each other in the outer optimization phase. These updates are aggregated via an all-reduce operation into a single “outer gradient”, which is then applied via an outer optimizer (typically, Nesterov Momentum) to the current global parameters to get the new global parameters, which then form the starting point for the next inner optimization phase. The pseudocode appears in Algorithm 1 and visualization in Figure 1 . This approach compares favorably with standard data-parallel distributed training (in which each worker computes one gradient on a batch of local data, after which all gradients are aggregated into one and applied to the current parameters). See ( Douillard et al., 2024 ) for details. The benefit is that the total communication requirements goes down by a factor of the number of inner optimization steps (typically, 50-100) compared to standard data-parallel, leading to better running time. However, in certain settings such as cross-datacenter training, communication links between workers have low bandwidth, and workers are forced to block in each outer optimization phase until all the outer gradients are communicated before continuing computation. This leads to wasted time due to idle compute. The goal of this paper is to mitigate this issue by developing techniques to overlap communication with computation leading to better compute utilization. The particular approach studied here is to allow the communication of outer gradients to happen in parallel with the computation of the immediate next inner optimization phase. I.e., at the end of each inner optimization phase, workers dispatch the computed outer gradients to be communicated to the other workers, and then immediately start executing the next inner optimization phase without waiting for the outer gradient all-reduce to finish. The all-reduce operation is allowed as much as time as it takes for the inner optimization phase to complete. Thus, at the end of the inner optimizaiton phase, the all-reduced outer gradient from the previous inner optimization phase is available, and can be applied to the local parameters. A naïve implementation of the above approach leads to worse convergence than standard DiLoCo since the all-reduced outer gradients are applied with a delay of one entire inner optimization phase. To improve on this, we develop an eager version of this approach based on the following idea. Note that the local outer gradient computed at each worker is already available to that worker before the all-reduce with the other, non-local, outer-gradients. This local outer gradient can serve as a good proxy for the all-reduced outer-gradients, and can be used in an outer optimization step at each worker before starting the next inner optimization phase. Then at the end of that inner optimization step, when the all-reduced outer gradients become available to the worker, the (delayed) non-local outer gradients can be applied to the parameters, along with the new (fresh) local outer gradient. We call this method eager since it eagerly applies local gradients without waiting for the non-local gradients to arrive at the worker. Our experiments show that eager updates significantly help reduce the performance hit of naïve delayed outer gradients and achieve training loss close to standard DiLoCo. When factoring in compute utilization however, eager updates significantly improve over standard DiLoCo. We provide details in the following sections. 2 Algorithms Figure 1 : Data flow and operations in standard DiLoCo. Here, 4 workers execute in parallel and alternate sequentially computation (the outer and inner optimization steps) and communication (averaging outer gradients across workers). Figure 2 : Data flow and operations in DiLoCo with delayed outer gradients. Here, 4 workers execute optimization steps in parallel with each other, as well as with the communication required for averaging outer gradients. This is accomplished by delaying the application of the averaged outer gradient in the outer optimizer. In this section we describe the algorithms studied in detail. For all algorithms, we denote the model parameters as \theta . We use the superscript notation \theta^{(t)} to indicate the parameters at a given step t , and the subscript notation \theta_{m} to denote a particular shard of the DiLoCo replica. For example, \theta^{(t)}_{m} indicates the parameters of DiLoCo replica m at step t . If no subscript is used, the parameters are replicated across DiLoCo replicas. Note that it is possible for parameters to not be replicated and yet to be of the same value. 2.1 Standard DiLoCo DiLoCo is an instantiation of the FedOpt framework of Reddi et al. (2021) applied to language models which is a bi-level federated optimization paradigm using an inner optimizer , Adam ( Kingma and Ba, 2014 ) , and an outer optimizer , SGD with Nesterov momentum ( Sutskever et al., 2013 ) . The DiLoCo algorithm is shown in Algorithm 1 and visualized in Figure 1 . Algorithm 1 DiLoCo 1: M replicas 2: Synchronization frequency H 3: Model replicas \{\theta^{(0)}_{1},\dots,\theta^{(0)}_{M}\} 4: Data shards \{\mathcal{D}_{1},\dots,\mathcal{D}_{M}\} 5: Optimizers InnerOpt and OuterOpt 6: parallel for replica m=1\ldots M do 7: for step t=1\ldots T do 8: x\sim\mathcal{D}_{m} 9: \mathcal{L}\leftarrow f(x,\theta_{m}^{(t-1)}) 10: \theta_{m}^{(t)}\leftarrow\texttt{InnerOpt}(\theta_{m}^{(t-1)},\nabla_{\mathcal{L}}) 11: 12: if t\mod H==0 then 13: \Delta^{(t)}_{m}\leftarrow\theta^{(t-H)}_{m}-\theta_{m}^{(t)} 14: \Delta^{(t)}\leftarrow{\small\texttt{async-send}}[\frac{1}{M}\sum_{m=1}^{M}(\Delta^{(t)}_{m})] 15: \texttt{block-receive}[{\Delta^{(t)}}] 16: \theta_{m}^{(t)}\leftarrow\texttt{OuterOpt}(\theta_{m}^{(t-H)},\Delta^{(t)}) 17: end if 18: end for 19: end parallel for In DiLoCo, M local replicas perform, in parallel, H steps of the inner optimizer InnerOpt on a different subsets of the data (L3 to L5 in Algorithm 1 ). Every H steps, each replica computes an outer gradient \Delta_{m}^{(t)}=\theta_{m}^{(t-H)}-\theta_{m}^{(t)} (L7), a delta in the parameter space, and communicates it to all other replicas.

Read the complete text