A Study on Distributed Strategies for Deep Learning Applications in GPU Clusters
Md Sultanul Islam Ovi · 2025
Read A Study on Distributed Strategies for Deep Learning Applications in GPU Clusters by Md Sultanul Islam Ovi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12832.txt.
Introduction
Ovi wrote a solo study in twenty twenty-five that weighs three ways to train deep nets across groups of graph chips. Each way trades speed, memory use, and the cost of keeping chips in sync. He ran side-by-side tests on a fast chip set and on a cheaper chip set.
One way, which copies the full net to each chip, gave a two-to-three-fold speed gain. A shard way cut top memory by four-to-six-fold but paid more in talk time. A hub way that let chips drift saved time but lost truth in its scores. A fixed hub way lost its speed gain to sync costs.
Catalog record
| CREATOR | Md Sultanul Islam Ovi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Study on Distributed Strategies for Deep Learning Applications in GPU Clusters Md Sultanul Islam Ovi Abstract As deep learning models grow in size and complexity, training them efficiently on single GPUs becomes increasingly infeasible. This study investigates the effectiveness of several distributed training strategies—Distributed Data Parallel (DDP), Fully Sharded Data Parallelism (FSDP), and Parameter Server (PS) models—for scalable deep learning on GPU clusters. We conduct empirical evaluations across multiple models and datasets to assess trade-offs in memory usage, training time, GPU utilization, and model accuracy. Our results show that while FSDP reduces GPU memory usage by over 60%, it increases training time by up to 6× compared to DDP. In contrast, asynchronous PS training improves throughput but can lead to degraded accuracy due to stale updates. Through comprehensive analysis, we provide practical insights into the strengths and limitations of each strategy, offering guidance for selecting suitable methods based on system constraints and training objectives. I Introduction Deep learning has emerged as a cornerstone of artificial intelligence, fueling breakthroughs in diverse fields such as autonomous driving, medical diagnostics, financial modeling, and natural language processing. Its remarkable success is largely attributed to its ability to learn hierarchical representations from vast amounts of data. However, the effective training of deep neural networks comes at the cost of intensive computational demands. Graphics Processing Units (GPUs), known for their ability to execute massively parallel operations, have become indispensable in meeting these demands, significantly reducing the time required for training complex models [ 1 ] . As deep learning models scale in depth and parameter count, reaching hundreds of millions or even billions of parameters, the limitations of a single GPU in terms of memory capacity and compute throughput become evident. These constraints hinder the training of large models and restrict batch sizes, leading to inefficiencies in both resource utilization and training convergence. To address this, researchers turn to GPU clusters, which interconnect multiple GPUs to operate in parallel. However, this distributed setup introduces new challenges such as communication overhead, synchronization delays, memory fragmentation, and workload imbalance, necessitating the development of effective distributed computing strategies. This research explores several prominent strategies that enable scalable and efficient training across GPU clusters. Data parallelism, one of the most widely adopted approaches, tackles the challenge of large datasets by distributing data batches across multiple GPUs, each maintaining a replica of the model and aggregating gradients after local computation [ 2 ] [ 3 ] . For extremely large models that do not fit into the memory of a single GPU, model parallelism is employed by splitting the model architecture across different GPUs [ 4 ] [ 5 ] . The parameter server architecture facilitates centralized parameter management and coordination across worker nodes, offering scalability in large-scale settings [ 6 ] [ 7 ] [ 8 ] . Fully Sharded Data Parallelism (FSDP) pushes memory efficiency further by partitioning model parameters, gradients, and optimizer states across all GPUs in a cluster, allowing significantly larger models to be trained within the same hardware constraints [ 9 ] [ 10 ] . Additionally, federated learning introduces a decentralized paradigm where models are trained locally on distributed clients without sharing raw data, offering a privacy-preserving approach to collaborative learning [ 11 ] . The objective of this study is to conduct a detailed evaluation of these distributed training strategies, assessing their relative strengths and weaknesses in terms of scalability, memory usage, training speed, and model accuracy. By providing empirical insights and comparative analysis, this work aims to inform the design and deployment of deep learning systems in GPU cluster environments and contribute to ongoing advancements in distributed training frameworks. II Literature Review on Distributed Training Strategies The exponential rise in data availability and the increasing complexity of deep learning models have necessitated the development of distributed training techniques that can fully exploit the computational power of modern GPU clusters. Traditional single-GPU approaches quickly become inadequate when training models with hundreds of millions of parameters, due to limitations in memory bandwidth, compute power, and I/O throughput. Consequently, a rich body of literature has emerged exploring various strategies for distributing workloads across multiple GPUs to improve scalability, training efficiency, and resource utilization. Data parallelism has been one of the earliest and most widely adopted distributed strategies. It partitions the input data across multiple GPUs, each with a complete replica of the model, and performs local computations before synchronizing gradients. Krizhevsky et al. [ 1 ] demonstrated the effectiveness of this approach by significantly reducing the training time of convolutional neural networks through dual-GPU processing. However, synchronization during gradient aggregation introduces communication overhead, especially in large-scale settings. To address this, Shi et al. [ 12 ] proposed optimizations in MPI communication to reduce transfer latency, enhancing the scalability of data-parallel approaches. Later works have focused on optimizing collective operations such as all-reduce and overlapping communication with computation to further reduce synchronization delays. Model parallelism tackles the problem from the opposite direction by dividing the model itself across multiple GPUs. This is particularly useful for extremely large models that cannot fit into the memory of a single GPU. Dean et al. [ 13 ] explored this approach in Google’s large-scale distributed systems, distributing different layers of deep networks across compute nodes. However, this technique introduces significant inter-GPU communication, which can become a bottleneck in performance. Shazeer et al. [ 14 ] addressed this by proposing the sparsely-gated mixture-of-experts architecture, which activates only a small subset of model parameters per forward pass, thereby reducing communication and memory overhead. An evolution of model parallelism is pipeline parallelism , which organizes the computation of a neural network into a pipeline across GPUs. By processing different mini-batches simultaneously at different pipeline stages, this method improves utilization and minimizes idle time. Huang et al. [ 15 ] introduced GPipe, which incorporates automatic pipeline splitting and micro-batching to enable near-linear scaling across multiple devices. Their work demonstrated that pipelined execution could significantly shorten training time without sacrificing convergence or model accuracy. Hybrid parallelism combines the benefits of data and model parallelism by distributing both the data and different parts of the model across GPUs. This is particularly advantageous in scenarios involving heterogeneous hardware or varied batch sizes. Akintoye et al. [ 16 ] investigated hybrid approaches and demonstrated their effectiveness in adapting to different system architectures while maintaining scalability and training throughput. These methods are often implemented in large-scale deep learning frameworks such as Megatron-LM and DeepSpeed, where careful orchestration of compute and communication is key. Another well-established distributed training framework is the parameter server (PS) architecture, which separates computation and parameter management. Worker nodes calculate gradients based on local data and send updates to centralized parameter servers, which maintain the global state of model parameters. Li et al. [ 6 ] provided one of the foundational implementations of this architecture, showing its ability to scale across large clusters. Enhancements such as Stale Synchronous Parallel (SSP) [ 8 ] were introduced to balance the trade-off between update freshness and communication efficiency. More recently, Jiang et al. [ 17 ] proposed adaptive scheduling techniques to improve load balancing and resource allocation in PS systems. Federated learning offers a distributed training framework that prioritizes