GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training CCS: Computing methodologies Machine learning CCS: Computer systems organization Dependable and fault-tolerant systems and networks Conference: Make sure to enter the correct conference title from your rights co
Keyao Zhang, Yiquan Chen, Zhuo Hu, Wenhai Lin, Jiexiong Xu, Wenzhi Chen · 2025
Read GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training CCS: Computing methodologies Machine learning CCS: Computer systems organization Dependable and fault-tolerant systems and networks Conference: Make sure to enter the correct conference title from your rights co by Keyao Zhang, Yiquan Chen, Zhuo Hu, Wenhai Lin, Jiexiong Xu, Wenzhi Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07035.txt.
Description
Keyao Zhang and five co-authors speed up the checkpointing that protects large language model training from crashes. Periodically saving a model lets training resume after failures, but standard saving pauses or slows GPU computation while data moves to the CPU and storage. Their system, GoCkpt, spreads one checkpoint transfer across multiple training steps so saving overlaps with useful work.
Each step transfers part of the checkpoint state plus some gradient data. Once the transfer finishes, the partial states are updated on the CPU into one consistent version, avoiding the inconsistency that multi-step transfer would otherwise cause. A transfer strategy maximizes both GPU-to-CPU bandwidth and SSD throughput. In tests GoCkpt raised training throughput by up to about thirty-eight percent over standard asynchronous checkpointing, and cut interruption time sharply.
Catalog record
| CREATOR | Keyao Zhang, Yiquan Chen, Zhuo Hu, Wenhai Lin, Jiexiong Xu, Wenzhi Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07035