Distributed Training under Packet Loss

Erez Weintraub, Ron Banner, Ariel Orda · 2025

Read Distributed Training under Packet Loss by Erez Weintraub, Ron Banner, Ariel Orda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07114.txt.

Introduction

Erez Weintraub, Ron Banner, and Ariel Orda make model training work over lossy networks in this 2025 paper. Top models train across thousands of graphics chips, often spanning data centers. Standard frameworks assume perfect links, and the constant confirmations slow things down.

Dropping the confirmations cuts delay, but lost packets can skew the math and stall learning. Their framework trains over unreliable links while keeping the gradient sums unbiased. Drift in the model values stays bounded, and users need not change their model code. The work opens training to cheaper, looser networks. It fills a gap that growing model sizes made urgent.

Catalog record

CREATORErez Weintraub, Ron Banner, Ariel Orda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distributed Training under Packet Loss Erez Weintraub, Ron Banner, Ariel Orda Abstract Background. State-of-the-art language and vision models are routinely trained across thousands of GPUs, often spanning multiple data-centers, yet today’s distributed frameworks still assume reliable connections (e.g., InfiniBand or RoCE). The resulting acknowledgment traffic and retransmissions inflate tail latencies and limit scalability. Leveraging unreliable connections will reduce latency but may sacrifice model’s accuracy and convergence once packets are dropped. A principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss has previously been missing. Results. We address this critical gap by introducing a novel distributed training framework capable of operating over unreliable connections, offering unbiased gradient aggregation and bounded parameter drift without modifying model code or optimizers. The key insight is a two-stage defense against missing messages: (i) Unbiased gradient aggregation —each worker reconstructs a consistent gradient estimate from whatever packets arrive, guaranteeing expectation-level correctness; and (ii) Bounded-drift parameter broadcasts —we prove the inter-worker model discrepancy remains O ⁡ ( 1 ) O(1) even after arbitrarily many iterations, preventing the unbounded divergence typical of asynchronous setups. Analytical bounds are matched by experiments on LLAMA2 7B model with 64 GPUs: tolerating 10 % 10\% random packet loss yields ≤ 0.8 % \leq 0.8\% perplexity change. This work bridges the gap between communication-efficient datacenter protocols and the accuracy and generalization guarantees demanded by modern large-model training, enabling robust, high-throughput learning on commodity or wide-area networks. 1 Introduction Distributed training of large neural network models has rapidly become essential due to increasing model sizes and data volumes. State-of-the-art language models, for instance, often require scaling across hundreds or thousands of GPUs ( tithi2025scaling ; epoch2025trends ) . While parallelization strategies such as data, model, and pipeline parallelism combined with advanced optimizers (e.g., ZeRO-2) have addressed many scalability issues, communication overhead and synchronization remain critical bottlenecks ( jin2025megascalemoe ) . Conventionally, distributed training relies on reliable connection (RC) communication protocols that guarantee reliable packet delivery, resulting in significant overhead and high tail latencies as clusters scale ( gangidi2024roce ) . Recent research has started exploring relaxed reliability assumptions, leveraging loss-tolerant communication protocols to mitigate synchronization delays ( wang2024mlt ) . Such approaches demonstrate potential gains in training throughput, albeit at the risk of gradient or parameter inconsistencies due to packet loss. While gradient compression techniques significantly reduce bandwidth requirements ( jia2024sdp4bit ) , they typically presume reliable delivery; packet drops in these scenarios can severely degrade performance. Similarly, techniques that introduce bounded asynchronous, such as Stale Synchronous Parallel (SSP), relax synchronization constraints to manage slow workers and network jitter, yet they still rely fundamentally on consistent communication ( kim2025halos ) . A key challenge remains largely unaddressed: enabling efficient, large-scale distributed training under genuinely unreliable networking conditions without compromising convergence or final model accuracy. Specifically, quantifying the effects of unreliable packet delivery on model divergence and devising principled, robust compensation strategies are critical open issues. In this work, we address these gaps by developing a novel framework for scalable distributed training explicitly designed for unreliable networks. We introduce unbiased gradient aggregation methods resilient to packet loss, analytically quantify the resulting model divergence, and propose practical strategies to maintain drift within acceptable bounds. By doing so, we demonstrate that it is possible to leverage the throughput advantages of loss-tolerant protocols while still achieving stable convergence and high accuracy. Our analytical and empirical findings offer a path forward for robust distributed training under realistic network conditions, bridging an important gap in current large-scale learning frameworks. 2 Previous Work Distributed training methods have advanced significantly to address scalability and communication bottlenecks inherent in training large neural network models. Optimizers such as ZeRO-2 partition model states across GPUs and overlap computation with communication, reducing memory consumption and enabling the efficient scaling of extremely large models ( rajbhandari2019zero ) . Despite these innovations, traditional distributed training typically employs TCP for reliable communication, resulting in significant synchronization overhead and increased latency, particularly as cluster sizes expand. To mitigate these challenges, recent studies have explored using loss-tolerant communication protocols over UDP, accepting controlled packet loss to improve throughput. For example, the Loss-tolerant Transmission Protocol (LTP) was introduced to reduce synchronization delays by tolerating packet losses without significantly harming convergence ( chen2023boosting ) . Similarly, Differential Gradient Transmission (DGT) adopts a selective gradient-update strategy based on contributions to model accuracy, thereby providing robustness even under unreliable communication conditions ( yang2023DGT ) . Alongside protocol-level innovations, gradient compression techniques have significantly reduced the bandwidth requirements of distributed training. Methods such as Deep Gradient Compression (DGC) combine momentum correction, gradient clipping, and sparsification, achieving substantial reductions in communication overhead without compromising accuracy ( lin2017deep ) . Other approaches, including Top- k sparsification, quantization methods like QSGD, and low-rank approximations such as PowerSGD, further optimize the balance between communication efficiency and convergence quality ( evaluation2023gradient ) . However, most gradient compression techniques implicitly assume reliable gradient delivery, potentially limiting their effectiveness under packet loss scenarios. Moreover, synchronization strategies have progressively relaxed strict synchronous constraints. Stale Synchronous Parallel (SSP) methods introduce bounded asynchronous updates, accommodating slow workers and network jitter ( ho2013lazytable ) . Dynamic SSP further adapts the staleness threshold based on runtime conditions to improve training efficiency while still fundamentally depending on consistent network communication ( zhao2019dssp ) . Finally, efficient implementations of collective communication primitives, such as All-Reduce using ring or butterfly algorithms, have become essential to achieving contention-free, bandwidth-optimal parameter aggregation ( patarasuk2009bandwidth ; wiki_collective ) . Despite these advances, current approaches either compromise accuracy to handle packet loss or strictly enforce synchronization, restricting scalability under genuinely unreliable network conditions. The present work uniquely addresses this gap by developing a robust analytical and empirical framework designed explicitly for training stability and accuracy under lossy communication scenarios. 3 Model Formulation We train a model with parameter vector \theta\in\mathbb{R}^{d} split into N shards \theta=\bigl(\theta^{(1)},\dots,\theta^{(N)}\bigr), where shard \theta^{(j)} is owned by worker W_{j} . (1) Local gradient computation. At iteration t , each worker W_{i} samples a mini-batch \mathcal{B}_{t}^{(i)} and computes its local gradient on all shards: g_{t}^{(i)}=\nabla_{\theta}L\bigl(\theta_{t}^{(i)};\,\mathcal{B}_{t}^{(i)}\bigr), which we partition into N pieces g_{t}^{(i)}=\bigl(g_{t}^{(i,1)},\dots,g_{t}^{(i,N)}\bigr) , where g_{t}^{(i,j)}\in\mathbb{R}^{|\theta^{(j)}|} . (2) Gradient communication Workers exchange their gradient pieces over UDP. A transmission of g_{t}^{(i,j)} from W_{i} to W_{j} succeeds with probability 1-p . Denote the indicator s_{t}^{(i,j)}\sim\mathrm{Bernoulli}(1-p). Each W_{j} aggregates whatever shards it receives into an unbiased estimate: \hat{g}_{t}^{(j)}=\frac{\sum_{i=1}^{N}s_{t}^{(i,j)}\,g_{t}^{(i,j)}}{\sum_{i=1}^{N}s_{t}^{(i,j)}}\quad\bigl(\mathbb{E}[\hat{g}_{t}^{(j)}]=G_{t}^{(j)}\bigr). (3) Local update. Worker W_{j} updates only its shard: \theta_{t+1}^{(j)}=\theta_{t}^{(j)}-\eta_{t}\,\hat{g}_{t}^{(j)}. (4) Parameter broadcast. Immediately after updating, W_{j} broadcasts \theta_{t+1}^{(j)} to all other workers via unreliable connection. Let r_{t}^{(j,i)}\sim\mathrm{Bernoulli}(1-p) indicate a successful reception by W_{i} . Each W_{i} then sets \theta_{t+1}^{(i,j)}=\begin{cases}\theta_{t+1}^{(j)},&r_{t}^{(j,i)}=1,\\ \theta_{t}^{(i,j)},&r_{t}^{(j,i)}=0,\end{cases} so that W_{i} holds a potentially stale full model \theta_{t+1}^{(i)}=(\theta_{t+1}^{(i,1)},\dots,\theta_{t+1}^{(i,N)}) . Quantifying Model Drift Packet drops in parameter broadcasts introduce discrepancies between replicas of the same shard. We now formalize and bound this “drift.” Unlike conventional distributed training—where missed gradient updates can lead to unbounded divergence between models over time (i.e., a growing consensus problem)—our approach ensures that the expected distance between any two models remains bounded with time t . Specifically, we show that this discrepancy is O(1) , rather than increasing with t , due to the periodic resetting effect of successful broadcasts. Theorem 3.1 (Bounded Model Drift) . Under

Read the complete text