Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo Lee · 2025
Read Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems by Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08540.txt.
Introduction
Jihyun Lim and co-authors train neural networks faster by letting slow and fast machines work together, in a 2025 paper. Standard training keeps all machines in lockstep, so slow processors hold back fast ones, and many labs just leave their slower chips idle. Local training, where each machine works alone for stretches, could help if the drift between machines stays in check.
Their twist is to add a planned bias in which data each machine sees and how their models are merged. That bias tunes the fleet so slow CPUs and fast GPUs pull in the same direction. In one test, the method trained a standard image model with 2 CPUs and 8 GPUs up to 32 times faster than lockstep training, at nearly the same accuracy. The result gives mixed hardware a practical path into deep learning work.
Catalog record
| CREATOR | Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo Lee Abstract Most large-scale neural network training methods assume homogeneous parallel computing resources. For example, synchronous SGD with data parallelism, the most widely used parallel training strategy, incurs significant synchronization overhead when workers process their assigned data at different speeds. Consequently, in systems with heterogeneous compute resources, users often rely solely on the fastest components, such as GPUs, for training. In this work, we explore how to effectively use heterogeneous resources for neural network training. We propose a system-aware local stochastic gradient descent (local SGD) method that allocates workloads to each compute resource in proportion to its compute capacity. To make better use of slower resources such as CPUs, we intentionally introduce bias into data sampling and model aggregation. Our study shows that well-controlled bias can significantly accelerate local SGD in heterogeneous environments, achieving comparable or even higher accuracy than synchronous SGD with data-parallelism within the same time budget. This fundamental parallelization strategy can be readily extended to diverse heterogeneous environments, including cloud platforms and multi-node high-performance computing clusters. 1 Introduction Most large-scale neural network training methods assume homogeneous parallel computing resources. For example, synchronous SGD with data parallelism, the most widely used parallel training strategy, incurs significant synchronization overhead when workers process their assigned data at different speeds. Local SGD with periodic averaging, a foundational distributed optimization algorithm in federated learning, also suffers from the same limitation. While it reduces synchronization cost by decreasing the frequency of model aggregation, it still incurs substantial overhead at each aggregation point. Consequently, on modern GPU servers, relatively slower CPU resources often remain idle during training, even though CPUs are still powerful compute resources. Some prior works have proposed deep learning methods designed to exploit heterogeneous system resources. BytePS dedicates CPUs to summation operations only while GPUs compute gradients [ 10 ] . HetSeq introduces bias into synchronous SGD with data-parallelism to more effectively utilize heterogeneous GPUs [ 6 ] . DistDGL also utilizes both CPUs and GPUs to train large-scale graph neural networks. FusionFlow is a load scheduler that uses CPUs to assist slower GPUs [ 13 ] . Metis is another load-balancing method designed for distributed training on heterogeneous GPUs [ 22 ] . While all these methods effectively utilize CPU resources to improve the efficiency of synchronous SGD, they commonly overlook the potential of using alternative distributed optimizers, such as local SGD. These methods also commonly introduce bias into gradient approximation, however, they do not investigate its impact. In this study, we focus on how to utilize heterogeneous compute resources to efficiently train neural networks. Specifically, we develop a general parallel neural network training framework that minimizes synchronization costs across heterogeneous resources while achieving accuracy comparable to that of synchronous SGD with data-parallelism. To eliminate synchronization costs, we first adopt local stochastic gradient descent (local SGD) with periodic model averaging. We propose a system-aware adjustment of the number of local updates within each communication round, allowing faster resources to perform more updates than slower ones. We then inject carefully controlled bias into local SGD to compensate for the fewer updates performed by slower resources. The key principle behind our proposed framework is to inject bias into gradient estimation through data sampling and model aggregation. First, the framework monitors the training loss of each data sample and gives slow workers a higher probability of sampling high-loss data. This biased data sampling is expected to improve the quality of local models trained on slow resources , thereby accelerating the overall training process. Second, our framework adjusts the aggregation weights based on the number of local updates per commmunication round. As a result, local models trained for more iterations more strongly contribute to global model training . In this study, we demonstrate that these two independently introduced, system-aware biases enable more effective utilization of heterogeneous system resources compared to the conventional synchronous or local SGD. To evaluate the performance of the proposed parallel neural network training framework, we perform representative deep learning benchmarks. Our extensive empirical study shows that our system-aware biased local SGD effectively utilizes not only fast GPUs but also relatively slower CPUs, achieving the highest accuracy within a fixed time budget. We conduct a comprehensive empirical study comparing several realistic heterogeneous settings, such as parallel neural network training using two CPUs and eight GPUs. The proposed framework is readily applicable to many realistic heterogeneous systems including cloud computing and multi-node high-performance platforms. Our main contributions are summarized as follows. • Our study explores a novel local SGD-based approach for efficient parallel neural network training on heterogeneous compute resources. We empirically prove that unbalanced local SGD is a suitable optimizer for deep learning in heterogeneous environments due to its efficiency and flexibility. • We demonstrate that, across heterogeneous compute resources, the synchronization overhead of local SGD can be eliminated by system - aware adjustment of the number of local updates per round. This makes our method readily applicable to large - scale deep learning applications. • We also show that introducing well-controlled bias into local SGD can dramatically accelerate parallel neural network training while matching the accuracy of synchronous SGD with data-parallelism. 2 Background Problem Definition – In this work, we consider a minimization problem as follows. \displaystyle F(\mathbf{W})=\frac{1}{N}\sum_{i=1}^{N}f(\mathbf{W},\xi_{i}), (1) where N is the number of data samples and f(\cdot,\cdot) is a loss function (e.g., cross entropy). Typically, f(\cdot) is a non-convex and smooth function such as neural networks. Local Stochastic Gradient Descent – Local stochastic gradient descent (SGD) is a variant of SGD designed for communication-efficient model training. First it replicates the model to all workers. Then, each worker locally trains the received model using a numerical optimization algorithm such as mini-batch SGD. After a certain number of updates, the local models are averaged globally. These steps are called communication round . The update rule is written as follows. \displaystyle\mathbf{W}^{t+1}=\mathbf{W}^{t}-\eta\frac{1}{P}\sum_{i=1}^{P}\sum_{j=0}^{\tau-1}\nabla f(\mathbf{W}_{i}^{t,j}) (2) Local SGD repeatedly runs communication rounds until the global model converges. In modern high-performance or large-scale cloud computing platforms, all individual workers can access the entire dataset thanks to parallel file systems. As a result, the dataset is IID, and thus \mathbb{E}[\nabla f_{i}(\mathbf{W})]=\nabla F(\mathbf{W}),\ \forall i\in[P] . Biased Gradient Approximation – To efficiently solve the minimization problem shown in ( 1 ), optimizers such as SGD approximate the first-order gradient of F(\cdot) using randomly sampled data. Instead of a single data point (or sample), a small batch of data is typically used to make a practical trade-off between system efficiency and stochastic efficiency. When the batch is sampled based on a uniform random distribution, the approximated gradient is unbiased such that \mathbb{E}[\nabla f(\mathbf{W},\xi)]=\nabla F(\mathbf{W}) . In this study, however, we break this convention and explore how introducing bias into the gradient approximation can accelerate parallel neural network training. 3 Related work Local SGD under IID Settings – Some prior works provide a theoretical foundation for local SGD under IID settings. [ 21 , 8 , 12 ] . Recently, STL-SGD has been proposed, which accelerates local SGD by adaptively adjusting model aggregation period [ 20 ] . While these previous works consider local SGD under IID