All is Not Lost: LLM Recovery without Checkpoints
Nikolay Blagoev, Oğuzhan Ersoy, Lydia Yiyu Chen · 2025
Read All is Not Lost: LLM Recovery without Checkpoints by Nikolay Blagoev, Oğuzhan Ersoy, Lydia Yiyu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15461.txt.
Introduction
Nikolay Blagoev, Oğuzhan Ersoy, and Lydia Yiyu Chen present CheckFree, a 2025 method that rescues crashed language model training runs without any checkpoints. Training on scattered or rented machines cuts cost but invites node failures that delete whole layers of the model. Saving full copies or duplicating work, the standard cures, tax every run even when nothing fails.
CheckFree replaces a dead stage with a weighted average of its nearest living neighbors, at no extra storage or compute cost. A variant with out-of-order pipelining also covers the first and last stages, copying neighbor layers where needed. Tests on LLaMa models up to 1.5 billion parameters exercised the recovery under varied failure patterns. Cheap resilience could open serious training to loose, low-cost machine pools.
Catalog record
| CREATOR | Nikolay Blagoev, Oğuzhan Ersoy, Lydia Yiyu Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15461
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
All is Not Lost: LLM Recovery without Checkpoints Nikolay Blagoev, Oğuzhan Ersoy, Lydia Yiyu Chen Abstract Training LLMs on decentralized and wimpy computation nodes, e.g., multiple on-spot instances, lowers the training cost and enables model democratization. The inevitable challenge here is the churn of nodes due to failures and the operator’s scheduling policies, leading to losing a stage - a part of the model. The conventional approaches to recover from failures is to either use checkpointing, where periodically a copy of the entire model is sent to an additional storage, or redundant computation. These approaches yield significant communication and/or computation overhead even in non-failure cases and scale poorly in settings with large models. In this paper, we propose, CheckFree , an efficient recovery method where a failing stage is substituted by a weighted average of the closest neighboring stages. In contrast to the state of the art, CheckFree requires no additional computation or storage. However, because of the nature of averaging neighbouring stages, it can only recover failures of intermediate stages. We further extend our method to CheckFree+ with out-of-order pipeline execution to tolerate crashes of the first and last stages. Thanks to out-of-order pipelining, behaviour of those stages is mimicked by their neighboring ones, which allows CheckFree+ to recover them by simply copying the weights from the immediate neighbour. To be able to recover the (de)embedding layers, CheckFree+ copies those layers to the neighboring stages, which requires relatively small storage overhead. We extensively evaluate our method on LLaMa models of model sizes from 124M to 1.5B with varying failure frequencies. In the case of low and medium failure rates (5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant computation in terms of convergence in wall-clock time by over 12% . Both of our proposals can be run via our code available at: https://github.com/gensyn-ai/CheckFree . 1 Introduction Large Language Models (LLMs) have demonstrated an unprecedented ability to perform on a variety of language tasks. State of the art LLM training utilizes multiple Graphic Processing Units (GPUs) to parallelize and shard the training. The model is split across several GPUs (constituting stages), which communicate with the node servicing the previous layers and node servicing the next layers to execute the model in pipeline parallel (PP) fashion 1 1 1 Together all nodes of a distributed model is termed a pipeline. . Typically, pipelining is combined with data parallelism (DP) where several of these pipelines are trained in parallel on different data, synchronizing their gradients at the end of an iteration. Training such large models can take several months even on specialized high performance clusters Dubey et al. 2024 ; Yuan et al. 2022 . Recent works have proposed training models on cheaper GPUs, such as decentralized or cloud spot instances (preemptible GPUs) Jang et al. 2023 ; Thorpe et al. 2023 ; Ryabinin et al. 2023 . Due to the high training time, (hardware) failures during the training are inevitable Zhang et al. 2022 . Moreover, the number of failures is expected to be higher for training on such wimpy spot instances since the failures can also occur due to spot nodes becoming unavailable Thorpe et al. 2023 . In a pipeline, a stage can have partial nodes or all nodes failure. We omit the former scenario, since failing node can be trivially recovered by using the alive nodes of the same stage as the checkpoints. For example, when new nodes join, the most recent weights can be downloaded from a peer in the same stage, as per Jang et al. 2023 . In this work, we aim to design recovery methods for stage failures. This could happen when either (i) there is no DP and each stage is only run by a single node or (ii) all the nodes of the same stage fails simultaneously. The latter case is plausible for a real deployment via spot instances since such nodes of the same stage would be rented from the same geographic location (to minimize the DP communication overhead) Yuan et al. 2022 , and in the case of high demand on that location, all spot instances becomes unavailable simultaneously Thorpe et al. 2023 . To deal with stage failures during training, conventional approaches make use of checkpointing - periodically saving the model state to a non-volatile storage, which should remain reachable throughout the training Wang et al. 2023 . When a failure occurs, the state of the model is reverted to the previously stored one, thus losing several GPU hours of training time. A careful tradeoff is chosen between checkpointing frequency and training throughput, as checkpointing too frequently to a remote storage can slow down the training time Wang et al. 2023 . For example, LLaMa 70B requires roughly 520GBs when serialized and even on high bandwidth networks 2 2 2 Assuming high-bandwidth CPU-to-GPU bus and 500Mb/s connection. , would require upwards of 20 minutes per checkpoint, thus impacting training at high checkpointing frequency. Alternatively to checkpointing, Bamboo Thorpe et al. 2023 uses redundant computation across the pipeline to recover failures where each node stores and does redundant computation for the following stage (in addition to their assigned stage). Both recovery methods require significant overhead either in storage, communication or computation. In this work, we propose an efficient recovery method CheckFree . Our novel approach can recover, even when an entire stage has failed, without the need for checkpointing and without redundant computations, thus significantly outperforming the state of the art. To achieve this, we exploit layer stacking Agarwal et al. 2024 and LLMs; natural resilience to layer omission Elhoushi et al. 2024 . When a stage has crashed, we reinitialize it with a weighted average of its two neighbouring stages, on a new node. However, this strategy fails to recover the first and last stages without suffering from significant convergence drop. For those case, we utilize the idea of out of order pipeline parallelism Blagoev et al. 2025 , such that the second and second to last stage learn the behaviour of the first and last stage respectively. We refer to the extension that also recovers the first and last stages as CheckFree+ . A visualization of the different approaches is presented in Fig. 1 , and the comparison of their costs is given in Table 1 . ((a)) Checkpointing with external storage. ((b)) Redundant computation (and storage). ((c)) Our solution(s). Figure 1 : Three recovery strategies from stage failure. a) In typical checkpointing strategies, an external non-faulty storage is used, which periodically stores checkpoints. Upon failure, a new node receives stale weights from the storage and continues training. b)In decentralized training, redundant computation can be used to recover from failure of an entire stage Thorpe et al. 2023 . Each stage performs also the computations of the subsequent stage. When a new node joins, it can download the stage 1 weights from the previous stage. c) Our solutions CheckFree and CheckFree+ do not require a checkpointing or redundant computation. Instead, we use the weights of the two neighbouring stages to initialize the failed stage. CheckFree+ also recovers the first and last stages by copying the corresponding neighbouring stage which is trained to simulate the same behaviour. Table 1