Distributed Deep Learning using Stochastic Gradient Staleness

Viet Hoang Pham, Hyo-Sung Ahn · 2025

Read Distributed Deep Learning using Stochastic Gradient Staleness by Viet Hoang Pham, Hyo-Sung Ahn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05679.txt.

Introduction

Training deep networks across many machines without waiting for slow workers is studied by Viet Hoang Pham and Hyo-Sung Ahn in a 2025 machine learning paper. Bigger networks and datasets make training slow, and ordinary backpropagation forces processors to lock step with each other.

Their method merges data parallelism, where machines train on different data slices, with fully decoupled parallel backpropagation, where layers update without waiting. More data is processed per round and locking delays shrink. The authors prove convergence to critical points under stated conditions. Tests training a classifier on the CIFAR-10 image set support the efficiency claims.

Catalog record

CREATORViet Hoang Pham, Hyo-Sung Ahn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distributed Deep Learning using Stochastic Gradient Staleness Viet Hoang Pham, Hyo-Sung Ahn Abstract Despite the notable success of deep neural networks (DNNs) in solving complex tasks, the training process still remains considerable challenges. A primary obstacle is the substantial time required for training, particularly as high-performing DNNs tend to become increasingly deep (characterized by a larger number of hidden layers) and require extensive training datasets. To address these challenges, this paper introduces a distributed training method that integrates two prominent strategies for accelerating deep learning: data parallelism and fully decoupled parallel backpropagation algorithm. By utilizing multiple computational units operating in parallel, the proposed approach enhances the amount of training data processed in each iteration while mitigating locking issues commonly associated with the backpropagation algorithm. These features collectively contribute to significant improvements in training efficiency. The proposed distributed training method is rigorously proven to converge to critical points under certain conditions. Its effectiveness is further demonstrated through empirical evaluations, wherein an DNN is trained to perform classification tasks on the CIFAR-10 dataset. 1 Introduction Deep neural networks (DNNs) have emerged as a prominent approach in both academic research and industrial applications. They have demonstrated significant efficacy in addressing complex tasks, encompassing many areas ranging from image classification ( Li et al. 2021 ) , speech processing ( Mehrish et al. 2023 ) , to medical applications ( Harangi 2018 ; Dargan et al. 2020 ) and the robotics field ( Karoly et al. 2021 ) . The remarkable achievements of DNNs are largely attributable to their deeply layered architectures, which enhance the robustness and performance of models in practical implementations. Additionally, training a DNN typically requires a large amount of training data to ensure the model’s generalization. The incorporation of multiple hidden layers within the network, coupled with the reliance on extensive training datasets, inevitably results in increased computational demands and prolonged training time. Most training methods are based on the stochastic gradient descent (SGD) scheme ( Bottou et al. 2018 ) . At each iteration, a mini-batch of training data is randomly selected, and parameter updates are executed based on the gradient of the cost function corresponding to the selected mini-batch. While employing the cost function of a mini-batch instead of the full dataset increases the number of iterations required for convergence, it substantially reduces the computational time per iteration. Consequently, SGD is considerably more efficient compared to methods utilizing full gradient information. However, with a constant step size, SGD ensures convergence only to a neighborhood of critical points. The backpropagation (BP) algorithm ( Rumelhart et al. 1986 ) is the standard technique for calculating gradients of cost functions. It necessitates performing a forward pass and a backward pass before parameter updates. These passes must be executed sequentially, layer by layer, which introduces a bottleneck and represents a significant source of time inefficiency. To overcome the limitations of SGD and BP, various parallel and distributed computing strategies ( Diskin et al. 2021 ) have been proposed to facilitate more efficient and timely training processes. These strategies can be classified into model parallelism, data parallelism, pipeline parallelism, and hybrid parallelism. Model parallelism represents a parallelization strategy wherein a large neural network model is divided into multiple modules, enabling concurrent operations across these modules while utilizing the same mini-batch. This approach facilitates the acceleration of DNN training by assigning distinct parameters and computations to each module, corresponding to specific layers of the model. ( Huo et al. 2018 ) proposed the Decoupled Parallel Backpropagation framework, which mitigates the issue of backward locking by employing delayed error gradients. Extending this concept, ( Zhuang et al. 2022 ) introduced a fully decoupled scheme, leveraging delayed gradient techniques to address locking challenges in both forward and backward passes. In data parallelism, the training dataset is partitioned into subsets and distributed across multiple workers. Each worker maintains a complete replica of the DNN and computes the gradients based on its respective subset of training data. This approach increases the amount of training data processed during each iteration, thereby mitigating the adverse effects associated with the SGD. The global DNN model is constructed through the aggregation of model replicas and/or the gradients computed by the workers. This aggregation is typically implemented using two primary architectures: the parameter server architecture and the decentralized architecture. The parameter server architecture ( Song et al. 2021 ) is a centralized framework wherein all workers communicate with a server responsible for aggregating gradients and updating the model. In contrast, the decentralized architecture ( Assran et al. 2019 ) requires communication among neighboring workers. Pipeline parallelism divides the computational task, including both data and model components, into a series of processing stages. Each stage processes its input from the preceding stage and immediately forwards the results to the subsequent stage ( Lee et al. 2017 ; Kim et al. 2020 ) . However, the inherent dependency between stages imposes constraints on the scalability of this approach. Several studies incorporate elements of both data and model parallelism methods ( Song et al. 2019 ; Ono et al. 2019 ) , though these efforts primarily target specific neural network architectures without providing comprehensive analysis or generalizable insights. This paper aims to design a distributed training method that integrates the prominent advantages of data parallelism and model parallelism. Both modules in model parallelism and workers in data parallelism are treated as computational units. In our proposed framework, these computational units are collectively referred to as agents, following the terminology of multi-agent systems. In alignment with data parallelism, the training data is partitioned into multiple subsets and distributed across different groups of agents for parallel processing. These groups are termed data-groups. The communication among data-groups follows the decentralized architecture. Agents within each data-group collaborate to execute the fully decoupled parallel backpropagation algorithm ( Zhuang et al. 2022 ) , thereby enhancing computational efficiency during each iteration. Since multiple agents estimate parameters for the same part of the neural network model, they are organized into a model-group. A consensus scheme is applied to each model-group to ensure that the parameters of the corresponding model part achieve consistency across the agents. The main contributions of this paper are as follows: • We integrate data parallelism with the fully decoupled parallel backpropagation algorithm to design a distributed training method that accelerates deep learning without compromising accuracy. • We provide convergence analysis of the proposed method, demonstrating its convergence to critical points under common assumptions in deep learning. • We conduct experiments by training a deep neural network for a classification task to validate that the proposed distributed training method achieves significant speedup compared to centralized settings. The remainder of the paper is outlined as follows. Section 2 presents the background on deep learning and communication protocols in multi-agent systems. The algorithm is described in detail in Section 3. Convergence analysis is provided in Section 4. Section 5 reports the experimental results to assess the performance of the proposed training method, while Section 6 concludes the paper. Notations We use \mathbb{R} and \mathbb{R}^{m\times n} to denote the set of real numbers and the set of m\times n matrices, respectively. Let \textbf{1}_{n} be the vector in \mathbb{R}^{n} whose all elements are 1

Read the complete text