Exploiting Multicast for Accelerating Collective Communication CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Artificial intelligence CCS: Networks Transport protocols CCS: Software and its engineering Software performance
Chao Xu, Xu Zhang, Zihang Luo, Yuyan Wu, Guoxin Qian, Yufeng Yao, and CHIHYUNG WANG, Jingbin Zhou · 2026
Read Exploiting Multicast for Accelerating Collective Communication CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Artificial intelligence CCS: Networks Transport protocols CCS: Software and its engineering Software performance by Chao Xu, Xu Zhang, Zihang Luo, Yuyan Wu, Guoxin Qian, Yufeng Yao, and CHIHYUNG WANG, Jingbin Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22428.txt.
Introduction
Xu and colleagues present MultiWrite, a many-to-many link method that cuts collective communication delays for large model training and inference. Current methods rely on unicast writes, sending extra copies across links for many receivers. That waste clogs bottlenecks and slows end-to-end delivery. MultiWrite uses multicast ideas while fixing heavy management overhead and fit problems for AI workloads. On Ascend NPUs, its operators cut delays up to 33 percent on live deployed devices in long-term stress tests.
Catalog record
| CREATOR | Chao Xu, Xu Zhang, Zihang Luo, Yuyan Wu, Guoxin Qian, Yufeng Yao, and CHIHYUNG WANG, Jingbin Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60442 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22428
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploiting Multicast for Accelerating Collective Communication CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Artificial intelligence CCS: Networks Transport protocols CCS: Software and its engineering Software performance Chao Xu, Xu Zhang, Zihang Luo, Yuyan Wu, Guoxin Qian, Yufeng Yao, and CHIHYUNG WANG, Jingbin Zhou Abstract . Reducing collective communication latency is a critical goal for large model training and inference in both academia and industry. Many-to-many communications, such as AllGather and AlltoAll (dispatch), are core components of modern parallelization strategies. State-of-the-art implementations of these communications rely on unicast-based writes and transmit duplicate copies of the same data across physical links for multiple receivers. This redundant transmission congests network bottlenecks and degrades end-to-end latency. We present MultiWrite, a novel many-to-many transmission semantic that eliminates redundant packets to directly reduce operator latency. MultiWrite adopts multicast principles while addressing critical limitations of traditional multicast for AI workloads. These limitations include heavy management plane overhead and ecosystem compatibility issues. We implement MultiWrite on Ascend NPUs. Long-term stress tests demonstrate that our MultiWrite-based operators achieve up to 33% latency reduction on commercially deployed devices. 1. Introduction As large model parameter scales grow from hundreds of billions to trillions, distributed training and inference have become the foundational paradigm in AI (Artificial Intelligence) ( 25 ; 6 ; 44 ) . To support ultra-large models with limited hardware resources, the industry widely adopts data ( 18 ; 33 ) , tensor ( 35 ) , pipeline ( 12 ) , and expert ( 6 ; 17 ) parallelism to decompose computation. However, these strategies introduce frequent and intensive collective communication ( 26 ) between accelerators, such as GPUs and NPUs ( 41 ; 16 ) . Operations like AllReduce, AllGather, and AlltoAll now directly determine the scalability and end-to-end efficiency of distributed systems. Empirically, communication typically consumes 20-30% of runtime in large-scale training tasks, evolving from a supporting role to a core performance determinant. Among all collective communications, many-to-many patterns, such as AllGather and AlltoAll (dispatch), are particularly critical. In such transmissions, each node transmit duplicate copies of the same data to multiple receivers. When these copies traverse bottleneck links, communication latency grows significantly, and this in turn hurts training throughput and inference performance. Two trends have amplified this issue. First, communication domains are expanding rapidly ( 43 ) . The large EP technology introduced after DeepSeek V3 ( 5 ) allows AlltoAll domains to easily reach 64, 128, or even 256 nodes, creating more potential bottlenecks. Second, leading service providers now target TPOT (Time Per Output Token) below 5ms ( 38 ) . At this stringent requirement, even minor degradation in communication latency, which may account for 30% of end-to-end time ( 8 ) , can derail performance targets. Unfortunately, current approaches to addressing communication issues remain reactive and fragmented. Communication performance degradation is typically detected after training or inference jobs are running in production environments, and manifests as significant throughput losses or degraded end-to-end latency. Engineers then rely on vendor-specific custom diagnostic tools to inspect hundreds to thousands of devices across the cluster in order to pinpoint the exact problematic ports, links, or routing configurations. The root cause analysis may last from several days to several months ( 7 ; 20 ) . Even after root cause identification, fixes are ad-hoc: switching AlltoAll algorithms, adjusting routing weights, or adding communication-computation overlap. These tricks only work for specific hardware, parallel strategies, or model architectures. When workloads or clusters change, optimizations often break, forcing the entire cycle to repeat. This post-hoc firefighting paradigm is incompatible with the rapid iteration pace of AI. To address the issue of communication latency degradation caused by the transmission of redundant data on bottleneck links, we present the MultiWrite semantic based on our long-term experience and practice. The core idea behind MultiWrite is straightforward. Since the problem arises from redundant data traversing bottleneck links, and bottleneck links are inevitable in any network topology, we eliminate redundant data injection into the network entirely. Specifically, in many-to-many communication, only one copy of the same data will be sent to the downstream, regardless of how many destination nodes it needs to reach. Data is then replicated and forwarded at appropriate intermediate nodes to satisfy the packet reception requirements of all receivers. This technique is essentially a specialized form of multicast. Applying multicast to collective communication is not a novel concept, and prior works have proposed this idea. However, the industry has been reluctant to adopt this technology for several reasons, which are precisely the challenges we address in this work. The first challenge is the concern of whether multicast can really reduce latency. While multicast undoubtedly reduces bandwidth overhead on source egress links, but this does not necessarily mean multicast can reduce communication latency. Since collective communication transmissions typically traverse multiple links, bottleneck links may not be limited to the source node egress. Furthermore, traditional multicast implies that packet switching devices must support packet replication and forwarding, a capability that is absent from most current AI clusters, especially in scale-up domains. Second, the traditional multicast capability does not match the actual requirements of AI training and inference scenarios. As is well known, since DeepSeek v3, MoE (Mixture of Experts) and its corresponding large EP (Expert Parallelism) technology have become the de facto standard in large language models. The communication in MoE involves two all-to-all operations, also known as dispatch and combine. During the dispatch process, a single token of data needs to be sent to the top- k experts, where k is typically 8 or possibly more. The target experts are determined by the gate network. In this process, the destination nodes of two tokens emitted from the same source may be different, so these two tokens cannot be transmitted through the same multicast group. To meet the requirements of typical scenarios, the number of multicast groups would explode to a level of 10^{9} , which is simply unacceptable. Third, traditional multicast is incompatible with existing training and inference ecosystems. Under the hood, collective communication operators are implemented by decomposing them into multiple individual transfers, typically using unilateral write semantics. Traditional multicast requires additional operations for multicast group creation, management, and membership changes, as exemplified by IB (InfiniBand) UD (Unreliable Datagram) mode multicast implementation ( 14 ) . These operations have no place in current collective communication stacks. Integrating them would necessitate a full framework redesign, which is prohibitively expensive. MultiWrite addresses all three challenges. First, we identify and rigorously analyze non-niche scenarios where multicast-like techniques deliver measurable end-to-end latency gains, not just bandwidth reductions. Crucially, MultiWrite requires no changes to switches, which eliminates the primary obstacle to practical deployment. Second, we redesign multicast to carry destination information in packets rather than relying on pre-provisioned multicast groups, avoiding the group explosion problem in MoE scenarios. Third, we implement MultiWrite as an enhanced write semantic, not a separate module, preserving existing framework interfaces and usage patterns. We implement MultiWrite on Ascend NPUs and build optimized AllGather and AlltoAll operators on top of it. Long-term stress tests on commercially Ascend devices show 12%-33% latency reduction for these many-to-many operations. To summarize, this paper makes the following contributions: • From a practical AI infrastructure perspective, we identify