ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

Wenxiang lin 1 , Xinglin Pan 2 , Ruibo Fan 2 , Shaohuai Shi 1 , ∗ {}^{1,^{\ast}} , Xiaowen Chu 2 , 3 , ∗ {}^{2,3,^{\ast}} · 2026

Read ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training by Wenxiang lin 1 , Xinglin Pan 2 , Ruibo Fan 2 , Shaohuai Shi 1 , ∗ {}^{1,^{\ast}} , Xiaowen Chu 2 , 3 , ∗ {}^{2,3,^{\ast}} (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27844.txt.

Introduction

Lin and four colleagues introduce ZipCCL, a lossless library for LLM training. Training data, including activations, gradients, and parameters, often follows a near-Gaussian curve. ZipCCL uses exponent coding using the Gaussian curve of LLM tensors. It uses GPU-tuned compression kernels with traffic-aware data layout, and adaptive strategies switching operations by workload. On a 64-GPU cluster with mixture-of-experts and dense models, ZipCCL cuts traffic time by up to 1.35 times. It speeds end-to-end training by about 18 percent with no impact on quality.

Catalog record

CREATORWenxiang lin 1 , Xinglin Pan 2 , Ruibo Fan 2 , Shaohuai Shi 1 , ∗ {}^{1,^{\ast}} , Xiaowen Chu 2 , 3 , ∗ {}^{2,3,^{\ast}}
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52244
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training Conference: arxiv; 2026 Thanks: ∗ Corresponding author. Wenxiang lin 1 , Xinglin Pan 2 , Ruibo Fan 2 , Shaohuai Shi 1 , ∗ {}^{1,^{\ast}} , Xiaowen Chu 2 , 3 , ∗ {}^{2,3,^{\ast}} Abstract . Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication overhead, the potential of lossless compression has remained largely underexplored since compression and decompression typically consume larger overheads than the benefits of reduced communication traffic. We observe that the communication data, including activations, gradients and parameters, during training often follows a near - Gaussian distribution, which is a key feature for data compression. Thus, we introduce ZipCCL, a lossless compressed communication library of collectives for LLM training. ZipCCL is equipped with our novel techniques: (1) theoretically ground- ed exponent coding that exploits the Gaussian distribution of LLM tensors to accelerate compression without expensive online statistics, (2) GPU-optimized compression and decompression kernels that carefully design memory access patterns and pipeline using communication-aware data layout, and (3) adaptive communication strategies that dynamically switch collective operations based on workload patterns and system characteristics. Evaluated on a 64 - GPU cluster using both mixture-of-experts and dense transformer models, ZipCCL reduces communication time by up to 1.35 × \times and achieves end - to - end training speedups of up to 1.18 × \times without any impact on model quality. 1. Introduction As large language models (LLMs) continue to scale, training on massive GPU clusters has become inevitable ( 26 ; 42 ) . To fully utilize these resources, various parallelism strategies have been developed, including Data Parallelism (DP) ( 3 ) , Pipeline Parallelism (PP) ( 29 ) , Tensor Parallelism (TP) ( 3 ; 29 ) , Expert Parallelism (EP) ( 37 ; 12 ) , and Fully Sharded Data Parallelism (FSDP) ( 50 ) . While enabling parallel computation across thousands of devices, these methods introduce substantial communication overhead, which often becomes the bottleneck for system scalability. Existing efforts to mitigate this overhead fall broadly into three categories. The first category focuses on scheduling optimization , which overlaps computation with communication to mask latency ( 39 ; 24 ; 32 ; 14 ; 46 ) . The second category involves topology-aware operations , optimizing collectives based on specific network hardware properties ( 25 ; 26 ; 45 ) . The third category adopts communication compression , primarily utilizing lossy techniques such as quantization ( 40 ) or sparsification ( 15 ) . While effective in reducing data volume, these methods introduce irreversible information loss. For the expensive and complex training of large-scale LLMs, this lossy nature poses significant risks and quantification difficulties regarding model convergence and final accuracy. (a) Compression ratio. (b) Execution time. Figure 1 . Left: Compression ratio (original communication volume / compressed communication volume) averaged over training iterations; Right: Execution time of a lossless compressed All-Gather communication with different lossless compression methods for Llama3-8B with FSDP. Distinct from these works, we explore a promising yet underutilized direction: lossless compression . By leveraging data redundancy and entropy coding (e.g., Huffman ( 11 ) , ANS ( 6 ) ), lossless compression provides a bit-exact representation without any information loss. Our insight stems from the statistical properties of LLM training tensors, including activations ( 47 ) , gradients ( 38 ; 23 ) , and parameters ( 41 ) . Driven by the extensive use of Layer Normalization and well-conditioned optimization landscapes, these tensors typically follow Gaussian or log-normal distributions ( 21 ; 41 ; 20 ; 4 ; 19 ) (some experimental results can also be found in § 6.6 ). In the de-facto BFloat16 (BF16) format ( 17 ) , this statistical regularity manifests as a highly concentrated distribution of exponent values, where a small subset of exponents accounts for the vast majority of the data. To validate this potential, we analyze the communication patterns of a Qwen3-A22B ( 42 ) Mixture-of-Experts (MoE) model, specifically focusing on the bandwidth-intensive All-to-All operations required at MoE layers. As illustrated in Fig. 1(a) , tracking the communication volume over 40,000 training iterations reveals that the original data volume is 1.33\times larger than its losslessly compressed counterpart. Furthermore, this compression gain increases as training progresses, suggesting substantial potential for mitigating communication bottlenecks in large-scale runs. However, realizing this potential in practice is challenging. The core obstacle is that the computational overhead of compression often outweighs the bandwidth savings. While lossless compression has been applied in checkpoints ( 44 ; 10 ) or inference ( 48 ; 8 ) , applying it to the critical path of training communication requires ultra-low latency. General-purpose GPU compressors like DietGPU ( 16 ) and nvCOMP ( 30 ) are ill-suited for this regime. They fail to leverage communication-aware memory hierarchies and incur significant kernel-launch overheads. As shown in Fig. 1(b) , this leads to a “performance cliff”: despite reducing data volume, the substantial compression and decompression overheads make these methods slower than direct uncompressed communication (NCCL) for FSDP workloads, effectively turning a network bottleneck into a computational one. To address the issue, we present ZipCCL, a lossless communication collective library (§ 3 ) for LLM training, which bridges the gap between the theoretical compression potential of near - Gaussian LLM tensors and practical GPU performance with three key innovations: ❶ We propose a theoretically grounded exponent coding technique (§ 4.1 ) that exploits the normality of LLM tensors to bypass dynamic histogram construction. By deriving theoretical bounds on exponent concentration in normally distributed floating - point data, we directly compute a compact lookup table for exponent remapping, eliminating the need for online sorting and enabling deterministic, low - latency compression. ❷ We design GPU-optimized compression and decompression kernels (§ 4.2 ) with a communication-aware data layout, which is used to carefully design memory access patterns and pipelines. By leveraging the characteristics of communication operations, we design data layouts and access patterns that eliminate redundant data transformations. In the kernel implementation, we carefully account for global memory access costs and shared memory banking effects, and implement pipelined execution to overlap computation with data movement. ❸ We propose adaptive communication strategies for All-to-All operations (§ 5 ) in MoE model training under imbalanced expert computation loads, and we design a dynamic switcher for Reduce-Scatter that adapts to system characteristics. We implement ZipCCL as a drop - in replacement for standard NCCL collectives and evaluate it on a 64 - GPU cluster using representative MoE (DeepSeek - V3, Qwen - MoE) and dense (Llama3 - 8B) models. Experiments show that ZipCCL reduces communication time by up to 1.35\times and achieves end - to - end training speedups of 1.16 – 1.18\times over state-of-the-art training systems including Megatron-LM ( 29 ) and TorchTitan ( 22 ) . 2. Preliminaries and Motivations 2.1. Parallelism Paradigms Parallelism is an essential technique for large - scale LLM training. Common parallelism strategies include Data Parallelism (DP) ( 3 ) , Pipeline Parallelism (PP) ( 29 ) , Tensor Parallelism (TP) ( 3 ; 29 ) , Expert Parallelism (EP) ( 37 ;

Read the complete text