SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100 ​ k + 100\mathrm{k}+ GPUs

Jin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang · 2026

Read SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100 ​ k + 100\mathrm{k}+ GPUs by Jin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00357.txt.

Introduction

Lee and eight colleagues address fault tolerance for language model training on more than one hundred thousand graphics processors. Breakdowns there are routine, and restarts eat training time. Their SPARe scheme stacks spare data shards across parallel groups and reorders work on the fly to hide failed nodes during gradient sync. It matches classic replication for uptime while holding extra compute near steady, and at very large scale it cuts training time sharply against classic replication.

Catalog record

CREATORJin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60585
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00357

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100 ​ k + 100\mathrm{k}+ GPUs Jin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang Abstract In large-scale LLM pretraining systems with 100 ​ k + 100\mathrm{k}+ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe—Stacked Parallelism with Adaptive Reordering—a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only 2 ∼ 3 × 2\sim 3\times , even under high redundancy where traidional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize time-to-train. At extreme scale with up to 600 ​ k 600\mathrm{k} GPUs, SPARe reduces time-to-train by 40 ∼ 50 % 40\sim 50\% compared to traditional replication. 1 Introduction The industrial high-performance computing (HPC) system has scaled up to over 100\mathrm{k}+ GPUs to meet the computing demand of large-scale pre-training of AI foundation models. As the number of GPUs ( \#\mathrm{GPU} ) increases, the mean time between failures (MTBF) of system decreases as \mathcal{O}(\frac{1}{\#\mathrm{GPU}}) ( 35 ; 43 ) . Meanwhile, upon system failures, the restarting latency inflates quadratically or worse by default due to global collectives initialization and synchronization ( 33 ; 49 ) . As a result, cumulative waste of system downtime from frequent failures grows rapidly, leading to lower system availability 1 1 1 Fraction of time the system is operational: uptime divided by total wall-clock time ( 1 ) . and longer time-to-train 2 2 2 Total wall-clock time of training. Job Completion Time. . The LlaMa-3 paper ( 18 ) (Section 3.3.4) reports that the average failure rate is around 4 interruptions per 1000 servers (8 GPUs per server) per day. This corresponds to one failure every three hours on average for a 16\mathrm{k} H100-GPU system. Extrapolating using the empirically validated MTBF scaling model in 35 , failures would occur every 30 minutes at 96\mathrm{k} GPUs and every five minutes at 600\mathrm{k} GPUs. This projection is already harsh, yet still optimistic, as modern pretraining systems are increasingly heterogeneous, which further exacerbates failure rates ( 34 ; 32 ) . As failure-induced global restarts now happen once per hour or more, collective communication (re)initialization once neglected now dominates runtime. At 96\mathrm{k} GPUs, 49 show that \mathrm{NCCL\_init} and similar routines become a major bottleneck; even after optimization ( 33 ; 49 ) , key collectives still scale linearly with \#\mathrm{GPU} . 29 similarly report that large messages throughput in NCCL is best with linear-scaling algorithms. As a result, restart latency is projected to grow linearly with \#\mathrm{GPU} while MTBF shrinks inversely, pushing LLM pretraining system towards a restart-dominant regime where expected restart latency prevails over useful work time. For example, a 600\mathrm{k}+ cluster of 5 -min MTBF may spend \geq 60 -min on global restart after every failure. To address this challenge, three major approaches have been considered to achieve fault-tolerance in distributed training: checkpointing, partial recovery, and replication. Checkpointing attempts to minimize rework burden from progress loss by failures. Universal checkpointing ( 36 ) decouples checkpoint state from a fixed parallelism configuration to enable flexible and fast recovery. GEMINI ( 53 ) maintains lightweight in-memory snapshots for near-instant rollback. Just-in-Time checkpointing ( 25 ) adapts snapshot timing to high risk windows. Oobleck ( 31 ) reduces replay cost in pipeline parallelism ( 30 ) by replaying short segments. Multi-level checkpointing ( 17 ; 39 ; 38 ) amortizes I/O across device, host, and storage tiers. These advances lighten the rework burden once the system is back on the job . In the restart-dominant regime, however, the performance bottleneck shifts to restart downtime: the cumulative waste spent before the system can resume the job . In this setting, reducing the number of global restarts becomes critical to improving system availability hence reducing time-to-train. In parallel, traditional replication methods have been revisited to improve availability by masking failures with redundant computation, and partial recovery has been proposed to replace global restarts with cheaper, local system recoveries ( 37 ) . 21 disclosed that even minimal redundancy r=2 can tolerate a large number of failures and 3 proposed an optimized strategy to minimize time-to-train with checkpointing. However, replication is constrained by a fundamental ceiling: degree- r replication incurs r\times more computation, which becomes prohibitive in practice. This strongly motivates the need for a direct yet practical approach to improving availability without extravagant waste, especially as we are at the doorstep of the restart-dominant regime. We hereby propose SPARe: Stacked Parallelism with Adaptive Reordering, a failure-masking scheme as effective as traditional replication yet pays only near-constant overhead around 2\sim 3\times even for high redundancy like r\sim 20 . Instead of fully replicating computation, SPARe stacks shards of computation across synchronous Data Parallelism ( 56 ; 47 ) so that redundant workload can be minimized by adaptively reordering the stacks. Also, SPARe is orthogonal to progress loss rework hence can seamlessly merge with any checkpointing scheme. Our contributions in this work are: • We derive working formulas for the expected failure count and computation overhead SPARe endures and pays, and validate them with our simulation results. • We optimize SPARe with checkpointing (SPARe+CKPT) to find optimal redundancy r to minimize time-to-train. • Lastly, we provide discrete-event simulations using SimGrid ( 10 ) (a widely used large parallel system simulation toolkit) and with realistic system parameters for a 600\mathrm{k} H100 cluster to benchmark SPARe+CKPT against traditional replication with checkpointing (Rep+CKPT) and checkpointing only (CKPT-only), showing 40\sim 50\% gain in time-to-train. 2 Background 2.1 Synchronous Data Parallelism Fig. 1 shows synchronous Data Parallelism ( 56 ; 47 ) : the system consists of N\in\mathbb{N} identical GPU groups, and the total training data is divided into N equal-size shards with index i\in[N]:=\{0,1,\dots,N-1\} , each distributed to the group of the same index. Each group holds an identical copy of the entire model parameters; hence, group size is decided by the size of the model to train. Based on recent trends in scaling ( 18 ; 15 ) it would be reasonable to assume that the size is hundreds to a few thousands. Hence, in this work we set N\sim 10^{2-3} . Each group deploys a complex parallelism topology ( 48 ) , so a failed node (GPU) stops the entire group. Figure 1 : Synchronous Data Parallelism. At each training step, group i computes a partial gradient \mathbf{g}_{i} from its shard i . After all groups are done computing their own partial gradients, the system communicator triggers global all-reduce collectives to aggregate them into the full gradient \overline{\mathbf{g}}=\frac{1}{N}\sum_{j=1}^{N}\mathbf{g}_{i} , and distributes \overline{\mathbf{g}} back to all groups for model updates. Hence, all N types of \{\mathbf{g}_{i}\} must be collectible for the system to finish one training step: every single loss of any of the group stops training. 2.2 Node Failures

Read the complete text