Tensor-Parallelism with Partially Synchronized Activations
Itay Lamprecht † ‡ Asaf Karnieli † Yair Hanani † Niv Giladi ∘ * Daniel Soudry ‡ † Intel, Israel ‡ Department of Electrical and Computer Engineering - Technion, Haifa, Israel ∘ AWS AI Labs {itay.lamprecht, asaf.karniel, yair.hanani}@intel.com {ngiladi}@amazon.com {daniel.soudry}@gmail.com · 2025
Read Tensor-Parallelism with Partially Synchronized Activations by Itay Lamprecht † ‡ Asaf Karnieli † Yair Hanani † Niv Giladi ∘ * Daniel Soudry ‡ † Intel, Israel ‡ Department of Electrical and Computer Engineering - Technion, Haifa, Israel ∘ AWS AI Labs {itay.lamprecht, asaf.karniel, yair.hanani}@intel.com {ngiladi}@amazon.com {daniel.soudry}@gmail.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19645.txt.
Introduction
Itay Lamprecht, Asaf Karnieli, Yair Hanani, Niv Giladi, and Daniel Soudry report a way to train large language models with less data sent between chips. Their work, from 2025, deals with tensor-parallelism, where a model is split across devices and activations must be synced.
They call their method CAAT-Net, short for a communication-aware design for tensor-parallelism. Tests on a model with seven billion parameters cut tensor-parallel communication by up to half with no real drop in pretraining accuracy. Smaller models were also tested, and in some cases the loss score even got better. The team shows gains for both training and inference.
Catalog record
| CREATOR | Itay Lamprecht † ‡ Asaf Karnieli † Yair Hanani † Niv Giladi ∘ * Daniel Soudry ‡ † Intel, Israel ‡ Department of Electrical and Computer Engineering - Technion, Haifa, Israel ∘ AWS AI Labs {itay.lamprecht, asaf.karniel, yair.hanani}@intel.com {ngiladi}@amazon.com {daniel.soudry}@gmail.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tensor-Parallelism with Partially Synchronized Activations Itay Lamprecht † ‡ Asaf Karnieli † Yair Hanani † Niv Giladi ∘ * Daniel Soudry ‡ † Intel, Israel ‡ Department of Electrical and Computer Engineering - Technion, Haifa, Israel ∘ AWS AI Labs {itay.lamprecht, asaf.karniel, yair.hanani}@intel.com {ngiladi}@amazon.com {daniel.soudry}@gmail.com Abstract Training and inference of Large Language Models (LLMs) with tensor-parallelism requires substantial communication to synchronize activations. Our findings suggest that with a few minor adjustments to current practices, LLMs can be trained without fully synchronizing activations, reducing bandwidth demands. We name this “Communication-Aware Architecture for Tensor-parallelism” (CAAT-Net). We train 1B and 7B parameter CAAT-Net models, with a 50% reduction in tensor-parallel communication and no significant drop in pretraining accuracy. Furthermore, we demonstrate how CAAT-Net accelerates both training and inference workloads. 1 Introduction As Large Language Model (LLM) training continues to scale, often requiring the use of hundreds or even thousands of devices, the need for efficient distributed training and inference techniques is constantly growing. Tensor-parallelism [ 1 ] is a critical component in the training of many well-known LLMs, such as GPT-3 [ 2 ] , Llama [ 3 , 4 ] , and BLOOM [ 5 ] , enabling the efficient utilization of hardware resources to support models with billions of parameters. By partitioning weight tensors across multiple devices, tensor-parallelism allows each device to store only a fraction of the model’s weights during computation. This significantly reduces the memory footprint per device, making it feasible to train extremely large models on hardware with limited memory capacity. The efficiency of tensor-parallelism in reducing training memory has made it a cornerstone technique for distributed training. During inference, where the LLM response time is critical, distributing the computation over multiple devices using tensor-parallelism can significantly reduce latency. A large portion of the total time spent training or using LLMs is consumed by the communication overhead inherent in tensor-parallelism, which arises from the need to synchronize activation or gradient tensors. Specifically, synchronization is done by all-reduce operations, where tensors from all devices are summed into a single tensor, which is copied to all devices. This operation occurs multiple times in each forward and backward pass and constitutes a substantial fraction of the total workload time. A recent approach to addressing this issue is pipelining communications and computation [ 6 ] in such a way that the communication is overlapped with the computation and the footprint of communication is reduced. However, this approach is limited: when tensor-parallelism is expanded to use more devices, the compute workload per device decreases, while communication payload per device remains relatively constant [ 7 ] . This means that the relative cost of communication grows as the compute-to-communication ratio decreases. In extreme cases, communication time can overcome computation time, and thus dominate the training process. For these reasons, even the largest language models are typically trained with a tensor-parallelism dimension of 8 [ 4 ] , utilizing fast inter-node communications for the heavy all-reduce operations. Improving tensor-parallelism efficiency is even more important given that the growth in compute power exceeds the growth in communication bandwidth [ 7 ] . This trend should further expose communication time in large-scale training. Minimizing tensor-parallelism communication can enable better hardware compute utilization and reduce overall training costs, especially when extending tensor-parallelism across nodes. This is in line with the recent trend of building multi-node systems with high bandwidth communication. In traditional tensor-parallelism, the activation tensors after communication are identical on all devices, i.e, fully synchronized. In this work, we show that LLM training can converge without fully synchronizing the activation tensors in the all-reduce operation. This means that we allow activations to vary on different devices after communication. We show that without full synchronization, the current training practice needs to be slightly adjusted. Failing to do so leads to critical issues such as a mismatch between forward and backward passes and numerical issues, which often result in training divergence. We utilize this insight to present the partial channel–reduce operation, in which only a subset of the channels in the hidden dimension of the activation tensors is reduced. Unlike regular all-reduce, activations are not identical on all devices after the partial channel–reduce operation. We introduce Communication-Aware Architecture for Tensor-parallelism (CAAT-Net) — a new model architecture that is tailored for tensor-parallelism by utilizing partial channel–reduce to decrease communication overhead. While CAAT-Net has a smaller communication overhead compared to an identical model with full all-reduce, it does not have fewer parameters or lower total compute. We train a Llama2-7B [ 8 ] with partial channel-reduce over 90B tokens, and the 1.1B parameter TinyLlama model [ 9 ] with partial channel–reduce over 100B tokens. We show that there is no significant degradation in validation loss and zero-shot accuracy on multiple tasks, while reducing the communication payload by 50% . Additionally, we train multiple small 130M parameter models, to study the effects of the number of synchronized channels and tensor-parallel degree on accuracy. We show the speedup of our proposed method in various settings. In summary, our contributions in this paper are as follows: • We show that when using tensor-parallelism, LLMs can be trained without fully synchronizing activation tensors. • We propose CAAT-net, a novel architecture that significantly decreases communication traffic in training with tensor-parallelism by synchronizing only part of the activation tensors. • We show that in various settings, CAAT-Net achieves no significant drop in pretraining accuracy, while accelerating both training and inference. 2 Related Work The challenge of efficient training and inference on a large scale has attracted much attention, both in the engineering and research fronts, in close correspondence with each other. While most of the literature focuses on reducing data-parallel communication, there have been a few works that focus on tensor-parallelism. Our approach is orthogonal to all methods covered in this section. Pipelining computation and communications. Some methods accelerate training with tensor parallelism by overlapping communications and computation, [ 6 , 10 , 11 , 12 ] . In Domino Deepspeed, the training batch is split into smaller pieces, and data dependency is broken such that communication is not on the critical path and can be overlapped with computation. However, with a growing tensor-parallel dimension, the computation per device is reduced while the communication is not, so overlapping efficiency is reduced as well. Additionally, in Nvidia GPUs such as H100/H200, pipelining communication and computation comes with a cost — the Stream Multiprocessors (SM) which perform the calculation, are also needed for communication. When pipelining communication and computation, some SM cycles are spent on communication-related tasks instead of computation. Furthermore, while pipelining computation and communication can accelerate training and inference, our method reduces total communication, and can potentially improve other aspects such as power consumption. Compressed communication. Another approach is to address the communication overhead by compressing the communication itself at the tensor-parallel or pipeline-parallel stages [ 13 , 14 ] . This can be done using various methods, such as auto-encoders or sending the top-K elements of the tensors. While compression succeeded in other aspects of optimization, it has yet been demonstrated that compression works for tensor-parallelism, to the best of our knowledge. In fact, we observed severe degradation in model accuracy with even minimal compression using