Decoupled DiLoCo for Resilient Distributed Pre-training
Decoupled DiLoCo Team · 2026
Read Decoupled DiLoCo for Resilient Distributed Pre-training by Decoupled DiLoCo Team (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21428.txt.
Introduction
Arthur Douillard and 16 colleagues break the lock-step synchronization barrier in large-scale language-model pre-training with Decoupled DiLoCo. Where single-program-multiple-data training stalls all accelerators on any slowdown or failure, their design splits compute across independent learners doing local inner steps. Learners asynchronously send parameter fragments to a central synchronizer, which skips failed or straggling learners via a minimum quorum, an adaptive grace window, and dynamic token-weighted merging. Chaos-engineering tests show strictly zero global downtime with competitive text and vision performance.
Catalog record
| CREATOR | Decoupled DiLoCo Team |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94359 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21428
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoupled DiLoCo for Resilient Distributed Pre-training Decoupled DiLoCo Team Abstract Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this coupling, transient slowdowns, hardware failures, and synchronization overhead stall the entire computation, wasting significant compute time at scale. While recent distributed methods like DiLoCo reduced communication bandwidth, they remained fundamentally synchronous and vulnerable to these system stalls. To address this, we introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier and go beyond SPMD to maximize training goodput. Decoupled DiLoCo partitions compute across multiple independent “learners” that execute local inner optimization steps. These learners asynchronously communicate parameter fragments to a central synchronizer, which circumvents failed or straggling learners by aggregating updates using a minimum quorum, an adaptive grace window, and dynamic token-weighted merging. Inspired by “chaos engineering”, we achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime, while maintaining competitive model performance across text and vision tasks, for both dense and mixture-of-expert architectures. \uselogo 1 Introduction Modern LLM pre-training relies on the tightly coupled single program, multiple data (SPMD) paradigm (e.g., data, tensor, and sequence parallelism) that requires global synchronization at every step. This “monolithic” approach creates a major reliability bottleneck: a single hardware failure or straggler can stall the entire system. As compute scales, the sheer number of components transforms rare hardware failures into routine occurrences. This is exacerbated by the long duration of pre-training regimes, where frequent interruptions lead to significant downtime and wasted compute. Framing this analogously to the CAP theorem ( 13 ) , we argue the primary bottleneck of modern pre-training is a rigid adherence to parameter consistency. To reason about this trade-off, we define an analog of the “CAP properties” for the pre-training setting as follows: • Consistency (C): Every accelerator maintains a view of a globally synchronized set of model weights. • Availability (A): Training continues in the presence of hardware failures. • Partition Tolerance (P): Training continues despite interconnect instability or communication delays. Under this view, SPMD pre-training prioritizes consistency over all else: it sacrifices availability and partition tolerance to ensure every accelerator maintains a view of the global model state. While recent slice-granular “elastic” methods ( 30 ) can reconfigure the SPMD computation to run on a smaller subset of healthy accelerators, the global overhead of fault detection and cluster-wide resizing still incurs significant downtime, as illustrated in Figure 1 (top). Recent distributed alternatives like DiLoCo ( 25 ) and its streaming variant ( 23 ) have successfully addressed communication bottlenecks by reducing bandwidth requirements through intermittent synchronization. However, because previous iterations of DiLoCo remained fundamentally synchronous, they still enforced strict consistency across the cluster, leaving the system just as vulnerable to localized hardware failures and straggler penalties. Figure 1 : Slice-granularity elasticity vs decoupling : the system continues training with fewer “slices” of TPU chips when there is a localized failure. When a failure occurs affecting 1 of the M replicas, the decoupling approach allows the other \nicefrac{{M-1}}{{M}} replicas in the system to continue stepping. Inspired by the Pathways vision ( 6 ) , we argue that pre-training should prioritize availability and partition tolerance over consistency by moving beyond the present tightly-coupled paradigm. We propose Decoupled DiLoCo, a distributed training framework that decomposes a global cluster into independent, asynchronous “learners”. By evolving DiLoCo’s intermittent synchronization into a fully asynchronous communication protocol, we limit the “blast radius” of a hardware failure to a single learner. This allows the majority of the system to continue training uninterrupted during local failures or reconfigurations ( Figure 1 ). Through careful system and algorithm co-design, we demonstrate that maximum training goodput can be achieved in chaotic environments without sacrificing final model quality. Contributions Our core contributions are: • Decoupled DiLoCo: We introduce a distributed training framework that evolves previous bandwidth-focused methods by decomposing monolithic SPMD clusters into independent, asynchronous learners. This prioritizes availability and partition tolerance over strict consistency. • Scalable System Architecture: We design a system architecture featuring a central synchronizer to facilitate asynchronous parameter reconciliation, enabling the framework to operate efficiently at pre-training scales. • Empirical Validation: We provide extensive empirical evidence that Decoupled DiLoCo achieves comparable downstream performance to standard data-parallel training across various model and compute scales for both dense and Mixture-of-Experts (MoE) architectures on text and multi-modal evaluations. Moreover we show that pre-training with our asynchronous framework doesn’t hinder later post-training capabilities. • Robustness via Chaos Engineering: We apply chaos engineering ( 7 ) principles to LLM pre-training, demonstrating that our framework maintains high availability and model quality, even under aggressive, continuous hardware failures. 2 Preliminaries Let \theta^{(t)} denote the model parameters at step t . We wish to train a model across M compute clusters (called learners for brevity), each of which has its own model replica \theta_{m}^{(t)} , for m\in[M] . In SPMD data-parallel training, each learner computes a batched gradient, and a global gradient is aggregated across learners. This requires synchronization across the learners at every step, incurring potentially high amounts of communication and bandwidth. It also means that a slowdown in a single learner delays the entire computation, and a failure requires, in principle, a replay of the partially-completed step. In DiLoCo ( 25 ) , each of the M learners trains in parallel using an inner optimizer (e.g. AdamW) on distinct data shards, and only synchronize weights every H steps. Rather than averaging the weights, DiLoCo applies an outer optimization step ( 65 ) when t\bmod H=0 by treating differences in parameter space as gradient estimates: \displaystyle\Delta^{(t)} \displaystyle=\dfrac{1}{M}\sum_{m=1}^{M}\Delta_{m}^{(t)}=\dfrac{1}{M}\sum_{m=1}^{M}\left(\theta_{m}^{(t-H)}-\theta_{m}^{(t)}\right) (1) \displaystyle\theta^{(t)}_{m} \displaystyle=\texttt{OuterOpt}(\theta^{(t-H)}_{m},\Delta^{(t)}) We refer to \Delta_{m}^{(t)} as an outer gradient . 25 found that letting OuterOpt be SGD with Nesterov momentum ( 81 ) greatly improves model quality compared to weight averaging (which corresponds to an outer optimizer of SGD with learning rate 1). While DiLoCo reduces total bandwidth usage by a factor of H , it does not reduce peak bandwidth. This can be achieved by Streaming DiLoCo ( 23 ) . Partition the model weights \theta into P sets (called fragments ), and let \theta_{m,p}^{(t)} denote the p -th fragment held by learner m at step t . Let t_{1},t_{2},\dots,t_{P}\in\{0,\dots,H-1\} be distinct offsets. Then, at step t we only synchronize a fragment p if t\bmod H=t_{p} . Since ( 1 ) requires no actual gradient, the outer gradients can be computed fragment-wise. For t\bmod H=t_{p} , we compute \displaystyle\Delta^{(t)}_{p} \displaystyle=\dfrac{1}{M}\sum_{m=1}^{M}\Delta_{m,p}^{(t)}=\dfrac{1}{M}\sum_{m=1}^{M}\left(\theta_{m,p}^{(t-H)}-\theta_{m,p}^{(t)}\right) (2) \displaystyle\theta^{(t)}_{m,p} \displaystyle=\texttt{OuterOpt}(\theta^{(t-H)}_{m,p},\Delta^{(t)}_{p}) As written, ( 2 ) is blocking - the learners can’t continue training until they send \Delta_{m,p}^{(t)} and receive \Delta_{p}^{(t)} . However, 23 show that this communication and training can be overlapped - the learners can send \Delta_{m,p}^{(t)} and receive \Delta_{p}^{(t)} \tau steps later with little change in model quality for small values of \tau . Streaming DiLoCo offers a number of benefits, including significant reductions in bandwidth usage (total and peak) and the ability to make communication across steps asynchronous. However, Streaming DiLoCo still requires lock-step training across learners. Thus, when using Streaming DiLoCo with all-reduce, issues such as stragglers and learner failures can significantly slow down or halt training