MLP-Offload : Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25), November 16–21, 2025, St Louis, MO, USA DOI: 10.1145/3712285.3759864 ISBN: 979-8-4007-1466-5/2025/11 CCS: Information systems Data management systems CCS: Information systems Storage management CCS: Computing methodologies Distributed artificial intelligence Avinash Maurya, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae Abstract . Training LLMs larger than the aggregated memory of multiple GPUs is increasingly necessary due to the faster growth of LLM sizes compared to GPU memory. To this end, multi-tier host memory or disk offloading techniques are proposed by state of art. Despite advanced asynchronous multi-tier read/write strategies, such offloading strategies result in significant I/O overheads in the critical path of training, resulting in slower iterations. To this end, we propose MLP-Offload , a novel multi-level, multi-path offloading engine specifically designed for optimizing LLM training on resource-constrained setups by mitigating I/O bottlenecks. We make several key observations that drive the design of MLP-Offload , such as I/O overheads during the update dominate the iteration time; I/O bandwidth of the third-level remote storage tier remains unutilized; and, contention due to concurrent offloading amplifies I/O bottlenecks. Driven by these insights, we design and implement MLP-Offload to offload the optimizer states across multiple tiers in a cache-efficient and concurrency-controlled fashion to mitigate I/O bottlenecks during the backward and update phases. Evaluations on models up to 280B parameters shows that MLP-Offload achieves 2.5 × \times faster iterations compared to the state-of-the-art LLM training runtimes. † † cc-license: by 1. Introduction Large Language Models (LLMs) have revolutionized a broad range of domains thanks to text summarizing and knowledge distillation, enabling researchers to navigate complex scientific literature more efficiently. A natural evolution towards more general foundational models (FMs) capable of capturing complex correlations between different data modalities (e.g. using cross-attention) are beginning to unlock an even bigger impact on scientific progress. In a quest for advancing emergent behavior (capabilities not explicitly trained for but emerging spontaneously due to the massive scale and exposure to vast amounts of data during training), FMs’ scale and complexity continuously increase, requiring larger training infrastructures and incurring enormous costs. Pre-training and even fine-tuning were not even feasible without large-scale HPC systems: ChatGPT-3 was trained on over 10,000 GPUs ( Hamblen 2023 ) and cost over $4 million in each training session. Meta’s LLaMA model (released Feb. 2023), follows the same pattern: it used 2,048 Nvidia A100 GPUs to train on 1.4 trillion tokens, taking about 21 days ( Vanian and Leswing [n. d.] ) . Even worse, LLMs are rapidly growing both in terms of parameters and training data sizes. For example, ChatGPT-4 is estimated to be 10\times larger and was trained on 570\times more data than ChatGPT-3. Under these circumstances, the scale and cost required to train or fine-tune FMs become prohibitively expensive. Motivation: GPU Memory Wall A large number of state-of-the-art techniques have been proposed to make efficient use of the computational capabilities and massive parallelism offered by a large number of GPUs at scale. For example, DeepSeek ( Liu et al. 2024 ) has introduced several techniques in this direction. However, with GPU computations becoming more efficient, the bottleneck is shifting towards insufficient GPU memory capacities. For instance, the Gopher-280B model ( Rae et al. 2022 ) requires 4.8 TB ( Rajbhandari et al. 2021 ) , demonstrating terabyte-scale GPU memory requirements to train models in the order of hundreds of billions of parameters. This bottleneck leads to a so-called memory wall : the ever-increasing size of model parameters and auxiliary data structures such as activations and optimizer state (450 \times /2 years) far exceed the GPU memory growth of newer generations of devices (2 \times /2 years) ( Gholami et al. 2024 ; Thompson [n. d.] ; Maurya et al. 2024b ) . This trend, illustrated in Figure 1 is unsustainable. Furthermore, a majority of HPC datacenters need to serve multiple users running different workloads at the same time ( Chu et al. 2024 ) . Thus, the ability to train and fine-tune LLMs and FMs on a limited number of GPUs is highly desirable compared with the alternative of waiting for a long time in the batch queue to gain access to a large number of GPUs ( Duan et al. 2024 ; Jiang et al. 2024b ) . A figure illustrating the growth trend of large language models vs. the GPU memory, wherein the GPU memory grows at a much slower pace, necessitating the need for advanced memory optimizations and parallelization strategies to accomodate and run large models in democratized fashion. Figure 1. Model vs GPU memory growth. A figure illustrating the growth trend of large language models vs. the GPU memory, wherein the GPU memory grows at a much slower pace, necessitating the need for advanced memory optimizations and parallelization strategies to accomodate and run large models in democratized fashion. To address this challenge, state-of-the-art LLM training frameworks (e.g. DeepSpeed ( Rasley et al. 2020 ) and Megatron ( Shoeybi et al. 2019 ) ) introduce several techniques such as ZeRO redundancy optimization (e.g. partitioning the large-optimizer state across data-parallel replicas) ( Rajbhandari et al. 2020 ) , quantization, mixed-precision training ( Shoeybi et al. 2019 ) , etc. Still, this is not enough to alleviate the large GPU memory requirements. As a consequence, these techniques are often complemented with offloading of important data structures (model parameters, optimizer states) to the slower host memory (DRAM) ( Maurya et al. 2024c ) . When memory constraints are even tighter, the aggregated GPU and host memory capacity are not enough, prompting the need to further offload the data structures to third-level storage tiers (e.g., NVMe devices) ( Rajbhandari et al. 2021 ; Liao et al. 2024a ; Ren et al. 2021 ) . While the slower memory and storage tiers have significantly higher capacities, they also have significantly lower I/O bandwidth. For example, when pre-training a 20B LLaMA model without offloading (model parameters and optimizer states reside on aggregated GPU memory of a single node), the average iteration duration is 0.4 seconds. Offloading the optimizer states, used during the update phase, to the host memory increases the duration of the iterations to 3.7 seconds on average. Further offloading to a node-local NVMe increases the duration of the iterations to 67 seconds (§ 3.1 ). Such a large I/O bottleneck results in an overall 170 \times slowdown compared with training without offloading. At larger scales, such as training the 70B-parameter LLaMA across multiple nodes, communication overheads begin to dominate iteration time, which in turn amortizes the relative cost of NVMe offloading. As a result, the slowdown of NVMe-offloaded training relative to GPU-only training is reduced by about 7 \times , highlighting the interaction between communication and I/O bottlenecks in distributed settings. Therefore, despite the scale, reducing the I/O bottlenecks when using third-level storage is an important challenge that needs to be solved in order to be able to benefit from a significantly higher overall memory capacity without a prohibitive performance penalty. Limitations of State-of-the-art Existing advanced LLM training runtimes such as Microsoft’s DeepSpeed ( Rasley et al. 2020 ) , PyTorch’s Fully Sharded Data Parallel (FSDP) ( Zhao et al. 2023a ) , Colossal-AI ( Li et al. 2023 ) , etc. introduce novel asynchronous offloading techniques for both model and optimizer states. While FSDP allows only offloading to host memory, DeepSpeed and Colossal-AI runtimes enable NVMe offloading. Specifically, the model parameters and optimizer state are partitioned into model shards, which are distributed among worker processes (one per GPU) and collaborate using model parallelism. In turn, each model shard is decomposed into smaller chunks, called “subgroups”, which are processed in combination with mixed-precision techniques one by one as follows. During the forward pass and backward pass, a copy of the FP16 parameters is used on the GPUs to compute the gradients of each subgroup. As the backward pass progresses, the gradients are flushed to host memory and from there to third-level storage in FP32 format. Then, the update phase is performed on the CPU to avoid excessive traffic between the host memory and GPU memory, which negates the computational speed-up of GPUs. During the update phase the FP32 model parameters, optimizer state and gradients of each subgroup are fetched from third-level storage to host memory, the CPU-based updates are performed, and then the updated model parameters and optimizer state are flushed back to the third-level storage, while at the same time an FP16 version is pushed to the GPUs. Efficient pipelining and asynchronous I/O techniques are used to overlap computations with transfers between the GPU memory, host memory and third-level storage using specialized engines such as DeepNVMe ( Ren et al. 2021 ) . However, this is not enough to alleviate the limited I/O bandwidth of third-level storage. Furthermore, compute nodes are equipped with many GPUs that share the same host memory and third-level storage, resulting in competition for I/O bandwidth that is not mitigated. Another limitation of state-of-art approaches is the unnecessary back-and-forth movement between the host memory and third-level storage due to a suboptimal strategy to handle gradients and to reuse subgroups already available in the host memory. Key Insights and Contributions In this work, we present MLP-Offload , a novel approach that aims to alleviate the aforementioned limitations of state-of-the-art offloading techniques that make use of third-level storage to increase the overall memory capacity and I/O bandwidth. Specifically, we leverage a key observation that external storage of HPC systems, e.g., parallel file systems (PFS) and object stores, are typically underutilized during LLM training (except for occasional checkpointing ( Zhang et al. 2022 ) ), therefore they can complement node-local NVMe storage to provide a significant boost of I/O bandwidth. Combined with concurrency control to mitigate contention for I/O bandwidth and a better reuse of cached subgroups from one iteration to another, these ideas enable MLP-Offload to achieve a 2.5 \times speedup over state-of-art approaches. We summarize the key contributions as follows: (1) We perform an in-depth characterization of resource utilization during different phases of training when the optimizer state is offloaded to SSDs. In particular, we highlight several fundamental observations for our proposal: the CPU memory, utilized as caching buffers for asynchronous data transfers, experiences thrashing at every iteration; the upscaling and offloading of gradients are redundant; modern PFS capable of delivering several GB/s worth of parallel bandwidth (e.g. using Infiniband) remains unused; CPU cores remain idle due to slow disk writes despite overlapping transfers with CPU-based optimizer updates (§ 3.1 ). (2) Based on the above characterization, we propose a series of design principles: unified multi-level, multi-path asynchronous offloading using virtual tiers; optimized virtual tier concurrency control for multi-path I/O; cache-friendly ordering of model subgroup processing; and delayed in-place mixed-precision gradient conversion during updates (§ 3.2 ). These design principles are complemented by an I/O performance model detailed in § 3.3 . (3) We present MLP-Offload , an open-source implementation of the design principles that integrates with existing state-of-the-art LLM training runtimes such as DeepSpeed and Megatron. Our implementation insists on low-level optimizations such as process-exclusive multi-thread-shared locking mechanism in libaio – an optimized asynchronous POSIX I/O library; and efficient non-sequential ordering of asynchronous prefetch and flush operations to/from the disk (§ 3.5 ). (4) We evaluate MLP-Offload in a series of experiments in which we pretrain up to 280B parameters model on 32 \times A100-40GB GPUs. Our approach accelerates both backward and update phases by 13.5 \times and 2.3 \times , respectively, and speeds up the end-to-end training by 2.5 \times as compared to the state-of-art DeepSpeed (§ 4 ). 2. Background and Related Work Figure 2. DeepSpeed ZeRO-3 showing (a) hybrid model and data parallelism; wherein (b) the model parameters, activations, gradients, and optimizer states are sharded into subgroups; and (c) the optimizer states are offloaded to node-local NVMe. A set of illustrations showing the sharding of model and optimizer states with DeepSpeed's ZeRO-3 approach. Subfigure (a) shows model and optimizer sharded across 4 GPUs when using data parallelism with ZeRO-3 without tensor or pipeline parallelism. Subfigure (b) zooms on the chunks of model and optimizer on each GPU to show parameters, gradients, activations, momentum, variance with their corresponding FP16/FP32 precisions. It then zooms in on each of the above entities being further sharded into subgroups. Subfigure (c) show the offloading of optimizer states on node-local disk. Data, Pipeline, and Tensor Parallelism Various parallelism techniques for data, pipelines, and tensors, have been widely adopted to accelerate the training of large models, such as Convolutional Neural Networks (CNNs) ( Zhao et al. 2024 ) , Deep Learning Recommendation Models (DLRMs) ( Zhang et al. 2019 ) , Large Language Models (LLMs) ( Zhao et al. 2023b ) , and Large Vision Models (LVMs) ( Jiang et al. 2024a ) . Data parallelism accelerates training by running multiple model replicas, each running forward and backward passes in parallel with different input mini-batches and synchronizing at the end of the iteration in the update phase to combine the patterns learned from all mini-batches. Pipeline ( Huang et al. 2019 ) and Tensor ( Cai et al. 2021 ; Shoeybi et al. 2019 ) parallelism techniques split large models across GPUs with limited memory capacities. While pipeline parallelism splits the model vertically by placing a subset of model layers on a given GPU, tensor parallelism performs horizontal partitioning by splitting the model across all available GPUs. The combination of data, pipeline, and tensor-parallelism, often termed “3D parallelism”, is used to effectively scale large-scale training across thousands of GPUs for CNNs, LVM, LLMs, DLRMs, etc. However, given the ever-growing large sizes of LLMs and their rapid adoption in various fields, optimizations beyond 3D parallelism are required to run in resource-constrained setups. ZeRO Redundancy Elimination State-of-the-art LLM training runtimes, e.g., PyTorch’s Fully Sharded Data-Parallel (FSDP) ( Zhao et al. 2023a ) , Microsoft DeepSpeed’s Zero Redundancy (ZeRO) ( Rajbhandari et al. 2020 ) , Colossal-AI ( Li et al. 2023 ) , use redundancy elimination techniques to remove redundant parts of the model and optimizer states across data parallel ranks to minimize GPU memory consumption. To this end, DeepSpeed, a widely used ( Workshop 2023 ; Song et al. 2023 ) LLM training runtime, proposes three stages for eliminating redundancy across data parallel ranks: ZeRO-1 splits the optimizer states; ZeRO-2 splits both the optimizer states and gradients; and ZeRO-3 splits optimizer states, gradients, and model parameters ( Rajbhandari et al. 2020 ) . As illustrated in Figure 2 (a), training a P parameters model on N GPUs with full redundancy elimination (ZeRO-3) leads to \sim\lceil\mathcal{O(P)}/N\rceil order of memory savings at the expense of 1.5 \times higher communication overheads ( Rajbhandari et al. 2020 ) . The ZeRO-3 technique partitions model states across GPUs, requiring frequent scatter-gather collectives to reconstruct layers on demand, significantly increasing communication costs. Consequently, ZeRO-3, despite its memory efficiency, cannot be seamlessly combined with pipeline parallelism, which relies on efficient inter-stage communication. Instead, ZeRO-3 employs a hybrid model and data parallelism strategy, where training on N GPUs typically results in N “virtual” data-parallel replicas. Unlike conventional data parallelism where the model is fully replicated, these replicas remain virtual because model states are dynamically fetched and synchronized across GPUs as required. Mixed Precision Training Mixed precision training, proposed by Baidu and Nvidia research ( Micikevicius et al. 2018 ) , is another widely adopted approach to improve throughput and reduce the memory footprint in LLM training. This is illustrated in Figure 2 (b) in the left dotted block. Specifically, mixed precision uses two different copies of the model parameters, one in FP16 (or BF16 ), used to run the forward and backward passes, and another master copy in high-precision ( FP32 ), used by the optimizer in the update phase to retain higher stability ( Lee et al. 2024 ) . The activations and gradients produced by the forward and backward passes are in half-precision formats, leading to faster communications and faster computations. The low-precision FP16 gradients are upscaled to FP32 and used by the optimizer to perform the update phase. Several real-world LLMs, such as BLOOM-176B ( Workshop 2023 ) , OPT-175B ( Zhang et al. 2022 ) , GPT-3 ( Brown et al. 2020 ) , and GLM-130B ( Zeng et al. 2022 ) , are pre-trained using mixed-precision, thereby demonstrating the stability and efficiency of the mixed-precision approach. Sharded Model and Optimizer States Into Subgroups To reduce the intermediary memory required during computations, DeepSpeed’s ZeRO-3 shards the model parameters and optimizer states of each rank/GPU into subgroups, as depicted in Figure 2 (b). The subgroup sharding technique is unique to the DeepSpeed runtime and is unavailable on other runtimes such as FSDP or Colossal-AI. The size of these subgroups, M , is user-defined and specifies the total number of parameters per subgroup. The subgroups are evenly distributed among the GPUs. When using mixed precision for training, for each subgroup of M parameters, the forward and backward passes operate on the FP16 parameters and FP16 gradients, while the update step operates on the M corresponding FP32 parameters and FP32 optimizer state. This subgroup-style sharding allows for efficient piecewise computation and communication overlaps while minimizing memory footprint, as explained in ZeRO-Infinity ( Rajbhandari et al. 2021 ) . Optimizer State Offloading The optimizer state, held in FP32, is much larger than the model parameters and is only required during the update step. Thus, in memory-constrained scenarios, offloading it to the host memory (and further other multi-level tiers such as node-local NVMe) is a practical choice. The use of subgroups further facilitates swapping between the host memory and other multi-level tiers. This is shown in Figure 2 (c). When optimizer offloading is enabled, updates are typically performed on the CPU because (a) GPU memory is typically fully utilized by FP16 model parameters, and (b) transferring FP32 optimizer states and gradients between GPU and CPU, even subgroup-by-subgroup, incurs high I/O overhead. To efficiently manage these subgroup updates, each GPU worker allocates an independent I/O buffer on shared host memory, with enough capacity to hold a configurable number of subgroups. These I/O buffers serve two primary purposes: (1) caching FP16 gradients during the backward pass, upscaling them to FP32, and asynchronously flushing the FP32 gradients to disk for consumption during the update step, and (2) offloading to disk using asynchronous swap-in/out of optimizer subgroups between the host memory and the disk. Accelerating Model and Optimizer Offloading Several recent efforts focus on mitigating the I/O bottlenecks encountered due to offloading for training, fine-tuning, and inference scenarios. For instance, Fuyou ( Liao et al. 2024a ) and LoHan ( Liao et al. 2024b ) perform pipelined overlapping activation, parameter, and optimizer swapping across GPU-CPU-SSD to accelerate fine-tuning for extremely constrained scenarios, wherein neither model nor optimizer fits on the GPU memory; and is therefore not applicable for the scenario targeted in our work. Smart-Infinity ( Jang et al. 2024 ) uses computational storage devices (NVMe attached to FPGAs) to mitigate I/O between the NVMe and CPU, which accelerates the update phase but requires specialized NVMe devices. To mitigate the slowdown due to the expensive update phase, asynchronous update techniques such as one-step-delayed updates had been previously proposed in ZeRO-Offload ( Ren et al. 2021 ) : it overlaps the next iteration on the GPUs (with stale parameters) with the still ongoing CPU-based updates. However, this optimization was later removed from the runtime due model inconsistencies ( Team 2022 ) . DeepSpeed TwinFlow ( Wang et al. 2023b ) and Deep Optimizer States ( Maurya et al. 2024c ) are complementary efforts that exclusively focus on optimizing CPU-only offloading and do not optimize disk-based offloading. 3. Our Contribution: MLP-Offload 3.1. Gap Analysis Figure 3. Fraction of time spent in disk I/O during the update phase A graph showing the breakdown of iteration time, wherein 99\% of time is spent in I/O operations and only 1\% in training computations, when the optimizer is offloaded to the disk. A graph showing the I/O bandwidth of the local disk vs parallel file system for increasing number of competing processes within a node. This demonstrates that while the aggregated throughput remains the same, the read/write access latency increases when scaling processes. Figure 4. I/O bandwidth of SSD (local) vs. parallel file system (remote) A graph showing the I/O bandwidth of the local disk vs parallel file system for increasing number of competing processes within a node. This demonstrates that while the aggregated throughput remains the same, the read/write access latency increases when scaling processes. A graph showing the read and write I/O bandwidth fluctions for different subgroups, with mean read and write throughputs at 3.68 GB/s and 1.44 GB/s, respectively. Figure 5. I/O bandwidth under concurrency per subgroup using a local SSD A graph showing the read and write I/O bandwidth fluctions for different subgroups, with mean read and write throughputs at 3.68 GB/s and 1.44 GB/s, respectively. The combination of redundancy elimination and multi-tier offloading enables the pre-training and fine-tuning of large LLM models that normally would not fit in the aggregated GPU memory. However, the differences in I/O bandwidth between GPU memory, host memory, and third-level storage are large, especially when considering node-local SSDs or other forms of disk storage ( Ren et al. 2021 ) . As a consequence, I/O overheads may significantly delay especially the update phase, during which the offloaded subgroups need to be fetched, updated and flushed back. Training Iterations are Dominated by the Update Phase As a first step, we study the breakdown of a training iteration in terms of time spent in the forward pass, backward pass and the update phase, in order to check whether the update phase occupies a significant fraction of the overall iteration duration, which would validate the hypothesis that accelerating the update phase would accelerate the overall training iteration. To this end, we study DeepSpeed ( Rasley et al. 2020 ) , configured to use redundancy elimination and multi-tier offloading that includes a local SSD. We run experiments on a single node with 4 \times H100-80GB GPUs (Table 1 ), and record a breakdown of the iterations. We indeed confirmed that the iteration is dominated by the update phase. For instance, in the case of a 40B model (setup and methodology described in § 4.1 , respectively), the forward and backward passes finish in 0.6s (0.02%) and 28s (11%) of the total 242s iteration time, while the update phase finishes in 213s (89%). Thus, accelerating the update phase is an important step towards a faster end-to-end runtime. Update Phases are Dominated by I/O when using SSD Offloading We evaluate four model configurations, i.e., 20B, 40B, 70B, and 120B parameters model (Table 2 ). For the 20B model ( Wang et al. 2023a ) , the full FP32 parameters and optimizer state fit in the 512 GB host memory; therefore, we use it as a baseline. Larger model sizes offload their subgroups to an SSD. As shown in Figure 5 , in the case of the 20B model, the update phase completes \sim 30 \times faster than in the case of larger models that offload the model subgroups to an SSD. Specifically, when subgroups are offloaded to the SSD, 99% of the update phase duration is spent in SSD I/O, despite DeepSpeed’s optimized DeepNVMe engine that overlaps I/O with CPU computations. Thus, we can conclude that accelerating the fetches and flushes of offloaded subgroups to/from the host memory will significantly accelerate the update phase, which according to the previous experiments, will significantly accelerate the overall iteration. I/O Bandwidth of Different Third-Level Storage HPC systems are not only equipped with node-local SSDs but also external storage such as parallel file systems (PFS) and object stores (e.g. DAOS). Although external storage is shared by all compute nodes, and therefore they are subject to I/O competition, at a medium scale, it can provide a significant aggregated I/O bandwidth boost to complement the limited I/O bandwidth of node-local SSDs. To quantify this opportunity, we measure the raw read and write throughput of both node-local NVMe storage and remote PFS using microbenchmarks. Remote storage can offer higher throughput than node-local NVMe in some cases (e.g., Testbed-2 in Table 1 ). Additionally, given that each GPU process independently issues fetch and flush operations for its optimizer state subgroups, understanding I/O performance under contention is crucial. Figure 5 shows that while overall read/write throughputs remain constant as the number of concurrent processes increases, per-process latency (shown using lines and minor y-axis) worsens due to contention within the storage subsystem. This suggests that shared NVMe bandwidth saturation is a limiting factor, making I/O the primary bottleneck at scale. I/O Bandwidth under Concurrency during Offloading Finally, we analyze the effective read/write throughput perceived by the training runtime when offloading the optimizer states of a 40B model to a node-local NVMe. Figure 5 reveals oscillations in throughput due to runtime memory constraints, which restrict active subgroups in host memory to three at a time: one prefetched, one actively updated, and one flushed back to disk. The slow flush-back rate results in cases where the next subgroup is prefetched before the previous one is fully written, leading to intermittent spikes in read-throughput. However, the aggregated read and write throughput remains bottlenecked by NVMe’s write bandwidth, reinforcing the I/O limitations observed in prior experiments. 3.2. Design Principles Unified Multi-level, Multi-path Asynchronous Offloading using Virtual Tiers Leveraging local SSDs for offloading is scalable because the SSDs can be leveraged independently using model parallelism to achieve a high aggregated I/O bandwidth. However, model subgroups offloaded to local storage, such as SSDs, introduce high I/O overheads that dramatically slow down the update phase, as discussed in § 3.1 . On the other hand, external storage has significant potential to complement the limited I/O bandwidth of node-local SSDs and is typically under-utilized during pretraining (save for occasional checkpoints) ( Xian et al. 2024 ) . It is this observation that we capitalize on to extend existing multi-level offloading techniques with support for multi-path I/O at each level. Specifically, we unify all alternative storage (local SSDs, parallel file systems (PFS), object stores) into a virtual, third-level tier that can be used to offload model subgroups from the host memory. Then, based on an I/O performance model (detailed in § 3.3 ), we assign the subgroups to the alternative storage tiers proportionally to their I/O bandwidth. Using this approach, we can parallelize I/O operations such that slow tiers finish roughly at the same time as fast tiers because they store fewer subgroups. This I/O load balancing allows our approach to avoid any of the tiers becoming a bottleneck, thereby maximizing the acceleration of I/O operations for the overlapping with the computations. Although we illustrate this principle for the third-level tier, it can be generalized to any level (e.g., second-level tier to combine GPU-CPU or GPU-GPU HBMs, DDR, CXL, etc.). Optimized Virtual Tier Concurrency Control for Multi-Path I/O To enable high scalability for model parallelism, we allow each worker process (typically attached to a single GPU), to apply I/O load balancing on alternative storage independently of other workers. Using this approach, we avoid the need for expensive global synchronization. However, each compute node features multiple GPUs, which means that multiple processes will compete for a shared I/O bandwidth to alternative storage tiers. Therefore, we introduce a lightweight concurrency control strategy at the node level that allows only one worker process on each compute node to access a given alternative storage at a time. With this restriction of exclusive access per alternative storage, the full I/O bandwidth is guaranteed to be available to a worker process, which achieves lower latency (Figure 5 ) and I/O load balancing. At the same time, the other worker processes are free to compute updates for prefetched groups using all the available CPU cores, and use other alternative storage(s) in parallel, resulting in a natural interleaving that achieves global I/O load balancing. Note that a worker is not limited to accessing an alternative storage using a single I/O thread; it can leverage the preferred I/O parallelism of the alternative storage (e.g., a PFS is faster when using multiple I/O threads ( Gossman et al. 2024 ) ). Furthermore, if the preferred I/O parallelism cannot be saturated by a single worker, then the alternative storage can be subdivided further into multiple virtual tiers. Cache-Friendly Ordering of Model Subgroup Processing An important observation during the update phase of adaptive optimizers (e.g., ADAM ( Kingma and Ba 2014 ) ) is that the computations are embarrassingly parallel: each model subgroup has its own set of corresponding parameters, optimizer state and gradients that can be applied independently to obtain the new model parameter and optimizer state. Therefore, the order in which the subgroups are independently processed is inconsequential and does not impact accuracy or convergence . We exploit this observation to design a cache-friendly update strategy that leverages subgroups present on fast tiers (host memory) as much as possible. Specifically, in the first iteration, we start processing the subgroups in increasing order of IDs. After the update phase, the subgroups with a high ID will reside on fast (cache) tiers, while the subgroups with a low ID will reside on the slow tiers as they were evicted to slow tiers to make room for the subgroups with higher ID in the host memory. Then, in the second iteration, unlike state-of-the-art approaches, we reverse the order of processing the subgroups. This results in a significant acceleration of the update phase because a large number of subgroups with a high ID are already present on the fast tiers, thereby reducing I/O overheads due to memory thrashing. After the second iteration, the subgroups with low IDs end up on the fast tiers, which means we can take advantage of the increasing order again in the third iteration. Thus, we keep alternating between ascending and descending order of subgroup IDs which maximizes the benefits of caching. Delayed In-place Mixed-Precision Gradient Conversion during the Update Phase As mentioned in § 2 , offloading is typically implemented in combination with mixed-precision training. In this case, state-of-the-art approaches typically push the FP16 gradients that are computed during the backward pass from the GPU to the host memory, where they are converted to FP32. From there, they are flushed to the third-level virtual tier. However, during the update phase, each subgroup offloaded to the third-level virtual tier needs to be brought back to the host memory. Since the subgroup, composed of FP32 optimizer states (parameters, momentum, and variance), becomes even larger because of FP32 gradients, each fetch operation is slower than in the case when the subgroup only holds the optimizer state. On the other hand, FP16 to FP32 gradient conversions on a modern CPU have a high throughput (65 GB/s on Testbed-1 in Table 1 ) that is an order of magnitude larger than the fetch throughput. Thus, we propose an alternative strategy: during the backward pass, we simply store the FP16 gradients on the host memory– which anyway needs to reserve enough room for the FP16 gradients of all subgroups to enable gradient accumulation. Then, during the update phase, unlike state-of-the-art approaches, we fetch the subgroup (without FP32 gradients) and instead convert the FP16 gradients to their FP32 variant on-the-fly (using the same standardized numeric primitives as DeepSpeed ( Rasley et al. 2020 ; Wang et al. 2023b ) ). Thus, we accelerate both the backward pass (as we eliminate large asynchronous FP32 gradient flushes that can potentially delay the backward pass if they do not fully overlap with the computations) and the update phase (because the overhead of in-place FP16 to FP32 gradient conversion is typically negligible compared with the I/O overhead of fetching FP32 gradients from slow tiers). 3.3. Performance Model for Subgroup Allocation To enable load balancing for virtual tiers that can leverage multiple alternative storage options by aggregating multi-path I/O, we propose an approach that assigns model subgroups proportional to the I/O bandwidth of each alternative storage. Specifically, we assume each worker splits its model shard into M equally sized subgroups, which is typically the case to achieve computational load balancing. Furthermore, we assume a virtual tier composed of N storage tiers P_{i} , where 0\leq i