PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning

Yisu Wang, Ruilong Wu, Xinjiao Li, Dirk Kutscher * Hong Kong University of Science and Technology (Guangzhou) * Corresponding author is Dirk Kutscher, [email protected] · 2025

Read PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning by Yisu Wang, Ruilong Wu, Xinjiao Li, Dirk Kutscher * Hong Kong University of Science and Technology (Guangzhou) * Corresponding author is Dirk Kutscher, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18563.txt.

Introduction

Yisu Wang, Ruilong Wu, Xinjiao Li, and Dirk Kutscher speed up training of big neural networks spread across many machines. The machines must constantly average their gradient updates, and that traffic becomes the bottleneck. Squeezing the updates usually costs accuracy.

Their framework, PacTrain, prunes the network so many weights and gradients are zero to begin with. Because every worker knows the shared sparsity pattern, they can exchange only the nonzero values cheaply. Tests on vision and language models ran 1.25 to 8.72 times faster than rival compression systems. The scheme still fits the standard all-reduce routine clusters already use.

Catalog record

CREATORYisu Wang, Ruilong Wu, Xinjiao Li, Dirk Kutscher * Hong Kong University of Science and Technology (Guangzhou) * Corresponding author is Dirk Kutscher, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning Yisu Wang, Ruilong Wu, Xinjiao Li, Dirk Kutscher * Hong Kong University of Science and Technology (Guangzhou) * Corresponding author is Dirk Kutscher, [email protected] Abstract Large-scale deep neural networks (DNN) exhibit excellent performance for various tasks. As DNNs and datasets grow, distributed training becomes extremely time-consuming and demands larger clusters. A main bottleneck is the resulting gradient aggregation overhead. While gradient compression and sparse collective communication techniques are commonly employed to alleviate network load, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. This paper introduces PacTrain, a novel framework that accelerates distributed training by combining pruning with sparse gradient compression. Active pruning of the neural network makes the model weights and gradients sparse. By ensuring the global knowledge of the gradient sparsity among all distributed training workers, we can perform lightweight compression communication without harming accuracy. We show that the PacTrain compression scheme achieves a near-optimal compression strategy while remaining compatible with the all-reduce primitive. Experimental evaluations show that PacTrain improves training throughput by 1.25 to 8.72 × \times compared to state-of-the-art compression-enabled systems for representative vision and language models training tasks under bandwidth-constrained conditions. I Introduction Over the past decade, deep learning (DL) has seen significant growth driven by the development of increasingly complex large-scale models and massive datasets [ 1 , 2 , 3 ] . These DL models exhibit excellent performance in various computer vision (CV) and neutral language processing (NLP) tasks [ 4 , 5 , 6 , 7 ] . However, their immense scale makes large-scale distributed training (DT) time-consuming and expensive [ 8 ] . Large-scale DT incurs a high communication overhead, which is a challenge that spans large and small-scale setups. In large clusters with tens of thousands of GPUs provided by cloud providers ( e.g. , AWS, Azure, Alibaba Cloud) [ 9 , 10 ] , communication overhead increases linearly as the number of workers increases. Similarly, non-hyperscaler deployments, such as single- or multi-compute clusters in university labs with consumer-grade GPUs, often face communication bottlenecks owing to the constrained and highly variable bandwidths in such systems [ 8 ] . Gradient compression (GC) has emerged as a promising solution for mitigating this communication bottleneck by reducing the volume of the exchanged gradient data [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Most compression schemes ( e.g., , TopK [ 15 ] , TernGrad [ 13 ] , THC [ 14 ] , and DGC [ 16 ] ) are lossy and introduce some compression errors, meaning they often compromise both the convergence rate and attainable accuracy, and thus fail to accelerate the training process. Another potential approach for addressing communication bottlenecks in DDL is sparse collective communication (SCC). In recent years, practitioners in the deep learning community have observed that gradient sparsity often occurs in DNN training and have sought to exploit it [ 19 , 20 , 21 , 22 ] . For instance, OmniReduce [ 19 ] introduces a streaming aggregation to transfer sparse gradient blocks, and Zen [ 20 ] presents a hashing algorithm that leverages parallel GPU computing to achieve balanced communications. However, these approaches assume that gradients are sparse, which may not always be the case in practice. Nevertheless, as pointed out by prior studies [ 11 , 18 ] , a common problem of GC and SCC is that most schemes ( e.g. , DGC, OmniReduce, and Zen) are not compatible with all-reduce. All-reduce [ 23 , 24 ] is a highly scalable collective communication operation used to aggregate gradients among a set of DT workers. Another widely used gradient collection method is the parameter server (PS) that performs gradient collection on a centralized server [ 25 ] . All-reduce is inherently more parallel and scalable than PS based aggregation [ 11 ] . To address these challenges, we present PacTrain, a communication-efficient distributed training framework that combines model pruning and adaptive gradient compression. PacTrain can actively change the training gradient distribution through an unstructured pruning technique and leverages an efficient, non-lossy gradient compression scheme that is compatible with the all-reduce primitive to exploit gradient sparsity. However, transforming this high-level idea into an existing DT framework, such as PyTorch Distributed Data Parallel (DDP) [ 26 ] is not straightforward because of the abstracted communication interface. DDP is a widely used framework that accelerates training by distributing model replicas across multiple GPUs, synchronizing gradients, and averaging them to ensure consistency. Most DT frameworks, including DDP, offer communication hook functions that enable developers to customize gradient synchronization algorithms. However, these hooks typically expose only a reformatted one-dimensional gradient tensor, where gradient names are removed and gradient orders are rearranged. This abstraction complicates the establishment of a direct correspondence between the reformatted gradient tensor and the original model weights. Re-implementing the distributed training mechanism from scratch risks loses the performance optimizations inherent to the native frameworks. To address this, we propose a mask tracker mechanism that captures sparsity patterns within the reformatted gradient tensor, enabling efficient alignment and customization. By ensuring that all distributed training workers share global knowledge of gradient sparsity, we can use a pruning mask to reformat sparse gradients into low-dimensional dense gradient tensors. This approach enables lightweight gradient compression without compromising accuracy during the training process. Tab. 1 summarizes how the different methods of GC and SCC impact one or both factors that determine the Time-To-Accuracy (TTA): the iteration time and convergence speed compared with PacTrain. Compared to these methods, PacTrain can provide both fast convergence and compatibility with all-reduce, and it can improve TTA performance. We conducted extensive experiments on two datasets: CIFAR-10, CIFAR-100 [ 27 ] , using VGG19 [ 28 ] , ResNet18, ResNet152 [ 29 ] , and ViT-Base-16 [ 6 ] models. Our evaluations include specific link speeds of 100 Mbps, 500 Mbps, and 1 Gbps, chosen to reflect the WAN bandwidth in small multicluster scenarios. Our evaluation shows that PacTrain effectively minimizes TTA under bandwidth-constrained conditions. Our implementation demonstrates that PacTrain successfully adapts the compression scheme to prune-aware-distributed training scenarios and reduces TTA by up to 8.72 \times compared with native PyTorch DDP with variable-constrained network bandwidth. TABLE 1 : Impact of acceleration methods on training metrics. Method Conv. Speed Compatibility TTA PacTrain ✓ ✓ ✓ THC ✓ ✗ ✓ Terngrad ✗ ✓ ? DGC ✗ ✓ ? OmniReduce ✓ ✗ ✓ Zen ✓ ✗ ✓ Each method may positively (marked ✓) or negatively (marked ✗) affect performance. The performance of the methods marked with ? will vary according to the choice of the model architecture and training dataset. Compatibility denotes whether the method is compatible with all-reduce operation. This study makes the following contributions. Contribution 1. We propose that pruning can be used to enhance the gradient compression, thereby accelerating the process. Traditional pruning after training drops a subset of weights to accelerate inference and save memory size, but we show that the gradient sparsity enforcement (GSE) method based on pruning is crucial for realizing efficient distributed deep learning. Contribution 2. We propose that all

Read the complete text