DataStates-LLM : Scalable Checkpointing for Transformer Models Using Composable State Providers
Avinash Maurya 1 , M. Mustafa Rafique 2 , Franck Cappello 1 , Bogdan Nicolae 1 · 2026
Read DataStates-LLM : Scalable Checkpointing for Transformer Models Using Composable State Providers by Avinash Maurya 1 , M. Mustafa Rafique 2 , Franck Cappello 1 , Bogdan Nicolae 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16956.txt.
Introduction
Avinash Maurya, M. Mustafa Rafique, Franck Cappello, and Bogdan Nicolae speed saves for vast AI models. Big models now train on thousands of chips at once. Their state must be saved oft to guard the run. A crash with no save can waste weeks of work. Saves of such sprawl are slow and hard to steer.
Their 2026 preprint builds DataStates-LLM for those saves. State parts come from plug-in providers that stack. The stack serves data, tensor, and pipeline splits at once. Fast saves guard runs and let teams pause and probe. They also help trace why a run went wrong.
Catalog record
| CREATOR | Avinash Maurya 1 , M. Mustafa Rafique 2 , Franck Cappello 1 , Bogdan Nicolae 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16956
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DataStates-LLM : Scalable Checkpointing for Transformer Models Using Composable State Providers Avinash Maurya 1 , M. Mustafa Rafique 2 , Franck Cappello 1 , Bogdan Nicolae 1 Abstract The rapid growth of Large Transformer-based models, specifically Large Language Models (LLMs), now scaling to trillions of parameters, has necessitated training across thousands of GPUs using complex hybrid parallelism strategies (e.g., data, tensor, and pipeline parallelism). Checkpointing this massive, distributed state is critical for a wide range of use cases, such as resilience, suspend-resume, investigating undesirable training trajectories, and explaining model evolution. However, existing checkpointing solutions typically treat model state as opaque binary blobs, ignoring the “3D heterogeneity” of the underlying data structures–varying by memory location (GPU vs. Host), number of “logical” objects sharded and split across multiple files, data types (tensors vs. Python objects), and their serialization requirements. This results in significant runtime overheads due to blocking device-to-host transfers, data-oblivious serialization, and storage I/O contention. In this paper, we introduce DataStates-LLM , a novel checkpointing architecture that leverages State Providers to decouple state abstraction from data movement. DataStates-LLM exploits the immutability of model parameters during the forward and backward passes to perform “lazy”, non-blocking asynchronous snapshots. By introducing State Providers, we efficiently coalesce fragmented, heterogeneous shards and overlap the serialization of metadata with bulk tensor I/O. We evaluate DataStates-LLM on models up to 70B parameters on 256 A100-40GB GPUs. Our results demonstrate that DataStates-LLM achieves up to 4 × \times higher checkpointing throughput and reduces end-to-end training time by up to 2.2 × \times compared to state-of-the-art solutions, effectively mitigating the serialization and heterogeneity bottlenecks in extreme-scale LLM training. I Introduction Large Language Models (LLMs) have become a cornerstone of modern artificial intelligence (AI), driving unprecedented capabilities in text generation, comprehension, and domain-specific scientific discovery [ 6 ] . To achieve these capabilities, model sizes have exploded, routinely exceeding hundreds of billions to trillions of parameters [ 3 ] . Training such massive models requires High-Performance Computing (HPC) infrastructure comprising thousands of GPUs and involves running for weeks or months [ 34 , 6 ] . To manage the memory footprint of these models, training runtimes employ complex parallelism strategies: they combine Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP), along with optimizer state sharding techniques like ZeRO [ 23 ] and FSDP [ 36 ] . (e.g., BLOOM [ 34 ] and Llama 3 [ 3 ] ). Motivation: The Need for Scalable Checkpointing Given the large scale and extended duration of LLM training, checkpointing is a fundamental primitive necessary to ensure resilience and productivity. Hardware failures, software bugs, and timeouts are statistically inevitable at scale [ 30 ] . Without frequent checkpointing, the amount of lost computations that need to be recomputed is too costly both in terms of time and resources. For example, the Llama 3 405B model training involved 16K GPUs for 54 days and encountered failures every 2.8 hours [ 3 , 30 ] . Another example is Alibaba’s Unicron training that has a reported failure rate of 43.4% [ 8 ] . Beyond resilience, checkpoints are essential for addressing training instabilities (e.g., loss spikes in PaLM and GLM-130B [ 27 ] models), which are hard to predict and defend against, leaving rollback followed by adjustments as the most viable correction strategy. The productivity of several other prominent scenarios also depends on scalable checkpointing: reinforcement learning from human feedback (RLHF), transfer learning, faster convergence by merging different model states along the training trajectory. In such cases, the checkpoint frequency can be as high as every iteration. Regardless of the scenario, checkpointing involves the fundamental ability to capture a distributed AI model state frequently at scale without incurring significant overheads (i.e., blocking) that interrupt applications from making progress. Challenges and Limitations of State-of-the-Art Compared to conventional deep-learning models (ResNet, VGG, etc.), which range in hundreds of MBs and fit on a single GPU memory, LLMs are routinely composed of billions of parameters, leading to massive checkpoint volumes . The large model and optimizer states need to consistently combine multiple data structures of different data types and sizes , such as tensors, arrays, and custom Python objects (e.g., dictionaries, seeds of PRNGs, etc.). These data structures may be generated across different runtimes and languages, and may live in different memory tiers (e.g., Python dictionaries held in host memory, C++/CUDA tensors held in GPU memory, etc.). Furthermore, the combination of data-, pipeline-, and tensor-parallelism, along with redundancy elimination approaches (e.g., ZeRO [ 23 ] , FSDP [ 36 ] ) results in a fine-grain distribution of these data structures across a large number of compute nodes. We call these emerging multi-faceted variations in data types, data sizes, and different data sharding/distribution strategies across different storage tiers and compute nodes 3D heterogeneity of LLM checkpoints . This aspect is insufficiently addressed by state of art checkpointing approaches [ 31 , 12 , 20 , 15 , 33 ] , which results in significant overhead during checkpointing. Specifically, many checkpointing approaches implemented in state-of-the-art LLM training runtimes (e.g., DeepSpeed [ 23 ] ) deal with data sharding by initiating checkpoint capture in parallel from all GPUs to saturate I/O bandwidth. However, they do so in a blocking fashion, which interrupts the critical training path. Alternatives such as multi-level asynchronous checkpointing techniques aim to mask these overheads by capturing checkpoints on a fast tier, then flush the checkpoints from the fast tier to slower tiers in background [ 16 , 15 , 31 ] . However, it is not straightforward to adopt these techniques directly in LLM runtimes because of several reasons. First, GPUs don’t typically have enough spare memory capacity to capture full (CheckFreq [ 15 ] , GEMIMI [ 33 ] ). Second, although it is possible to capture the checkpoints directly on the host memory (e.g., TorchSnapshot [ 20 ] , CheckFreq [ 15 ] ), the limited GPU to host PCIe transfer bandwidth is orders of magnitude slower than GPU memory bandwidth and also needs to be shared by multiple GPUs typically co-located on the same compute node, thus incurring high I/O overheads. Without better techniques to capture a checkpoint on the initial fast tier, the benefit of multi-level asynchronous checkpointing is greatly reduced, to the point where it is not significantly faster than synchronous checkpointing. For example, despite the availability of high speed links (50+ GB/s network and 25+ GB/s PCIe), the LLM checkpointing throughput is far from saturating the link capacity (e.g., REFT [ 32 ] reports 38% saturation), and often drops as low as a few GB/s (e.g., as reported by Nebula [ 14 ] ). Heterogeneity of data generated by different runtimes and programming languages on different memory tiers is also a problem. The majority of checkpointing approaches mentioned above are highly optimized for tensor capture and serialization, but neglect other data structures. Therefore, training runtimes, such as DeepSpeed [ 23 ] , typically collect and capture high-level metadata and other Python data structures separately in a centralized fashion, before capturing the content of the distributed tensors in parallel. At scale, this blocking step becomes a significant bottleneck, even if the size of high-level metadata is significantly less than the size of