AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba, Artem Dzhalilov, Daniel Bobrov, Maxim Mashtaler, Alexander Gasnikov · 2025
Read AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models by Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba, Artem Dzhalilov, Daniel Bobrov, Maxim Mashtaler, Alexander Gasnikov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18182.txt.
Introduction
Nikolay Kutuzov and co-authors speed up training of large language models across many machines in a 2025 paper. Splitting training over a cluster saves time only if machines rarely sit idle. A known method lets each site train locally and sync now and then, but it wastes the variety of real hardware under shifting loads.
Their method, AdLoCo, works in three stages. Sites first run several light training streams in parallel and merge what they learn. Batch sizes then adapt on the fly to balance local computing against sync traffic, cutting wait times. Once batches grow past hardware-friendly sizes, a switch to gradient accumulation keeps training stable. The authors add a theory estimate for how many sync rounds full convergence needs.
Catalog record
| CREATOR | Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba, Artem Dzhalilov, Daniel Bobrov, Maxim Mashtaler, Alexander Gasnikov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba, Artem Dzhalilov, Daniel Bobrov, Maxim Mashtaler, Alexander Gasnikov Abstract Scaling distributed training of Large Language Models (LLMs) requires not only algorithmic advances but also efficient utilization of heterogeneous hardware resources. While existing methods such as DiLoCo have demonstrated promising results, they often fail to fully exploit computational clusters under dynamic workloads. To address this limitation, we propose a three-stage method that combines Multi-Instance Training (MIT), Adaptive Batched DiLoCo, and switch mode mechanism. MIT allows individual nodes to run multiple lightweight training streams with different model instances in parallel and merge them to combine knowledge, increasing throughput and reducing idle time. Adaptive Batched DiLoCo dynamically adjusts local batch sizes to balance computation and communication, substantially lowering synchronization delays. Switch mode further stabilizes training by seamlessly introducing gradient accumulation once adaptive batch sizes grow beyond hardware-friendly limits. Together, these innovations improve both convergence speed and system efficiency. We also provide a theoretical estimate of the number of communications required for the full convergence of a model trained using our method. 1 Introduction The rapid ascent of Large Language Models (LLMs) has turned them into the central tool of modern machine learning, yet their training remains an exceptionally resource-intensive process. Scaling Large Language Models (LLMs) across distributed clusters requires not only algorithmic innovation but also strategies for efficient utilization of heterogeneous hardware. While recent approaches such as DiLoCo ( Douillard et al. 2024 and classical algorithms such as LocalSGD ( Stich 2019 ) have demonstrated the potential of distributed optimization, they often utilize computational resources inefficiently under dynamic workloads, leaving efficiency gaps. In this work, we present a three-stage methodology designed to close these gaps and push the limits of distributed training. Our approach combines Multi-Instance Training (MIT), Adaptive Batching for DiLoCo, and a switch mode mechanism. MIT enables each compute node to host multiple lightweight training streams on different model instances, which are periodically merged to exchange knowledge. This not only maximizes hardware throughput but also reduces idle time in multi-node environments. Moreover, adaptive batching dynamically adjusts local batch sizes to balance computational workloads and provide for more efficient use of available resources, while also significantly diminishing synchronization overhead. Our adaptive batching framework is built up on the results shown in AdAdaGrad ( Lau et al. 2024b ). Given the foregoing, the natural interaction between MIT and adaptive batching yields a compounding benefit: as we approach the solution, fewer parallel instances remain active, but every instance uses a larger batch size, so the training process becomes more communication-efficient. To complement these techniques, the switch mode mechanism introduces gradient accumulation once adaptive batch sizes surpass hardware-friendly thresholds, ensuring robustness under batch-size limitations. Together, the three introduced components establish an effective synergy: multi-instance flexibility, memory-aware batching, and stability-preserving switching. We further support our methodology with a theoretical upper bound on the number of inter-instance communications required for convergence, yielding a rigorous understanding of its communication complexity. Our method achieves faster time-to-target perplexity compared to widely used distributed training baselines on industry standard benchmarks, while preserving convergence quality. These results highlight not only the practical value of our approach, but also its potential as a foundation for the next generation of scalable distributed LLM training. To facilitate reproducibility, we provide the implementation at https://github.com/funmagster/AdLoCo . 2 Preliminaries 2.1 Notation In this text, we denote the model’s parameters of the i -th instance at iteration k as x_{k}^{(i)} . The dataset is partitioned among M trainers belonging to \mathbb{T} , where the data shard \mathcal{D}_{i} corresponds to the i -th trainer. We use f(x_{k}^{(i)};\xi) to denote the loss function on a data sample \xi , with model of the i -th trainer at state x_{k}^{(i)} . Let B_{k} denote the mini-batch of samples used at iteration k with batch size b_{k} , and define the mini-batch gradient as: g_{k}=\nabla F_{B_{k}}(x_{k}):=\frac{1}{b_{k}}\sum_{\xi\in B_{k}}\nabla f(x_{k};\xi). (1) 2.2 Problem setting We consider the classical stochastic optimization problem in the distributed setting: \min_{x\in\mathbb{R}^{d}}F(x):=\mathbb{E}_{\xi\sim\mathcal{D}}[f(x;\xi)], (2) where f is a smooth, possibly non-convex loss function, and \mathcal{D} is the underlying data distribution. In the explored learning setting, the global objective equation 2 is optimized across training instances \mathbb{T} , each holding a local data shard \mathcal{D}_{i} . Each trainer performs local updates using its own data and periodically synchronizes the local model’s parameters with other trainers to reduce communication costs. 3 Related work The problem of efficient distributed optimization has received considerable attention in recent years, particularly in the context of large-scale deep learning. Classical distributed methods, such as LocalSGD Stich 2019 , reduce communication costs by allowing local computation between synchronization rounds, but they do not incorporate adaptive mechanisms to adjust training dynamics. Subsequent approaches like DiLoCo Douillard et al. 2024 introduced hierarchical communication patterns, further lowering synchronization overhead, yet they similarly lack adaptivity in batch sizing or coordination policies. On a different line of research, adaptive optimization methods such as AdAdaGrad Lau et al. 2024b explored variance-aware batch size adjustment strategies to improve convergence and computational efficiency. However, these methods are typically designed for single-worker or data-parallel setups without addressing distributed trainer coordination. 3.1 LocalSGD LocalSGD ( Stich 2019 ) is a commonly used distributed learning algorithm, which introduced the concept of federated periodic averaging. With 2 as the optimization objective, M parallel workers and data shards \mathcal{D}_{i} , at each step the i -th worker performs a local SGD step: x_{k+1}^{(i)}=x_{k}^{(i)}-\eta_{k}\nabla f(x_{k}^{(i)};\xi_{k}^{(i)}) (3) Instead of synchronizing after every step, as commonly seen in classic distributed learning algorithms, Stich 2019 proposed averaging the gradients only in every H steps. Specifically, if the next step is the averaging step ( k+1\ \text{mod}\ H=0 ), we perform the synchronization: x_{k+1}^{(i)}=\frac{1}{M}\sum_{i=1}^{M}x_{k}^{(i)} (4) After synchronization during step t+1 , the weights are changed according to 3 as usual, therefore, resulting in the complete LocalSGD’s weights update algorithm: x_{k+1}^{(i)}=\begin{cases}\displaystyle x_{k}^{(i)}-\eta_{k}\,\nabla f\!(x_{k}^{(i)};\,\xi_{k}^{(i)})\,,&\text{if }k+1\ \text{mod}\ H\neq 0,\\[10.00002pt] \displaystyle\frac{1}{M}\sum_{j=1}^{M}\bigl(x_{k}^{(j)}-\eta_{k}\,\nabla f\!(x_{k}^{(j)};\,\xi_{k}^{(j)})\bigr),&\text{if }k+1\ \text{mod}\ H=0\end{cases} (5) 3.2 DiLoCo DiLoCo is a distributed optimization algorithm, specifically, a variant of federated averaging. We assume that we have a base model (e.g., a transformer), that we want to train using a multi-instance procedure with its parameters denoted as x . In context of language modelling a series of tokens serves as the input, while the target is the input sequence that has been shifted by one token. Due to the multi-instance nature of the DiLoCo ( Douillard et al. 2024 ) algorithm the dataset is splitted into n data shards, where the i -th data shard is denoted as \mathcal{D}_{i} . The algorithm consists of two optimization processes denoted as outer and inner optimization, respectively. During H steps of the inner optimization stage each worker performs independently, calculating the gradients on its own data shard as seen in standard single-instance machine learning scenarios. During each of the T steps of the outer optimization stage, the gradients from each worker are gathered and averaged in the outer gradient. Therefore, every worker trains H\times T total training steps during the procedure. 3.3 AdAdaGrad AdAdaGrad ( Lau et al. 2024b ) is an optimization algorithm, which combines AdaGrad’s ( Duchi et al. 2011 ) learning rate adaptiveness with