TokenWeave : Efficient Compute-Communication Overlap for Distributed LLM Inference Conference: ; ;
Raja Gond, Nipun Kwatra, and Ramachandran Ramjee · 2025
Read TokenWeave : Efficient Compute-Communication Overlap for Distributed LLM Inference Conference: ; ; by Raja Gond, Nipun Kwatra, and Ramachandran Ramjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11329.txt.
Introduction
Raja Gond, Nipun Kwatra, and Ramachandran Ramjee speed up shared inference in twenty twenty-five. Running a large language model across many graphics cards splits the work, but the cards must talk to each other. That chatter can eat about a fifth of the time, even on fast links. One fix is to split work into small tasks and overlap talking with computing.
In practice, serving systems keep batches small for low delay, so splitting hurts more than it helps. The talking also steals processors from computing. TokenWeave is presented as the first system to overlap the two well in this setting. Readers who run model serving get a way to reclaim lost time on their hardware.
Catalog record
| CREATOR | Raja Gond, Nipun Kwatra, and Ramachandran Ramjee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TokenWeave : Efficient Compute-Communication Overlap for Distributed LLM Inference Conference: ; ; Raja Gond, Nipun Kwatra, and Ramachandran Ramjee Abstract Distributed inference of large language models (LLMs) can introduce overheads of up to 20% even over GPUs connected via high-speed interconnects such as NVLink. Multiple techniques have been proposed to mitigate these overheads by decomposing computations into finer-grained tasks and overlapping communication with subtasks as they complete. However, fine-grained decomposition of a large computation into many smaller computations on GPUs results in overheads. Further, the communication itself uses many streaming multiprocessors (SMs), adding to the overhead. We present TokenWeave to address these challenges. TokenWeave proposes a Token-Splitting technique that divides the tokens in the inference batch into two approximately equal subsets in a wave-aware manner. The computation of one subset is then overlapped with the communication of the other. In addition, TokenWeave optimizes the order of the layer normalization computation with respect to communication operations and implements a novel fused AllReduce-RMSNorm kernel that carefully leverages Multimem instruction support available on NVIDIA Hopper GPUs. These optimizations allow TokenWeave to perform communication and RMSNorm using only 2-8 SMs. Moreover, our kernel enables the memory bound RMSNorm to be overlapped with the other batch’s computation, providing additional gains. Our evaluations demonstrate up to 29% latency gains and up to 26% throughput gains across multiple models and workloads. In several settings, TokenWeave results in better performance compared to an equivalent model with all communication removed. The source code of TokenWeave will be released soon. 1. Introduction Figure 1 . Communication overhead of AllReduce for three models vs sequence length on 8xH100 DGX (error bar shows std. dev. across 5 runs). Even with NVLink/ NVSHARP, communication overheads can be over 20%. Figure 2 . Inference latency of Llama-3.3-70B on 8xH100 DGX for various sequence lengths. vllm-Multimem corresponds to vLLM with an optimized AllReduce implementation using Multimem and NVSHARP support. vllm-nocomm is a counterfactual baseline corresponding to only the computation time without any communication. The dotted lines represent relative performance compared to the vllm-Multimem baseline with TokenWeave providing up to 29% gains. Even at smaller sequence lengths, TokenWeave provides significant gains, e.g. 18\% at 1K , while prior schemes incur overheads. At sequence lengths \geq 4K , TokenWeave outperforms vllm-nocomm by not only recovering the complete communication overhead but also providing additional gains due to our fused AllReduce-RMSNorm kernel. Modern LLM deployments increasingly rely on distributed inference across multiple GPUs, driven by model size constraints or stringent latency requirements. Although high-speed interconnects like NVLink significantly improve communication bandwidth, substantial overheads remain. For example, in our experiments communication accounted for up to 23% of the end-to-end inference latency for Llama-3.3-70B ( Grattafiori et al. 2024 ) and Qwen2.5-72B ( Yang et al. 2024 ) models running on 8xH100 DGX (Figure 1 ). Solutions to mitigate the communication overheads rely on overlapping computation with communication. Flux ( Chang et al. 2024 ) , for instance, decomposes communication and computation operations into finer-grained tasks and fuses them into larger kernels to effectively hide communication latency. This method, however, requires complex kernel-level modifications, making it challenging to integrate into existing systems. NanoFlow ( Zhu et al. 2024 ) divides requests into nano-batches and co-schedules operations divided at operation granularity (FFNs, attention, collectives) to overlap resource (compute, memory I/O, communication) usage. However, NanoFlow requires intricate scheduling and substantial framework changes. A fundamental challenge with decomposing a large computation into multiple smaller computations is compute efficiency on modern hardware. Modern GPUs offer a very high degree of parallelism, which makes running a large computation kernel much more efficient than running multiple smaller kernels due to wave quantization effects. This problem becomes more severe as the size of the fine-grained computations reduces. As a result, to get non-trivial gains, most current approaches rely on batches with large number of tokens (upwards of 8k), so that the size of smaller computations do not become very small. Another problem with the overlap approach is that the communication kernel itself typically requires many SMs to perform the communication efficiently. These SMs are no longer available for the overlapping computation kernel which slows down the computation and eats into gains from the overlap. Due to these issues, as of this writing, none of the open-source serving systems like vLLM ( Kwon et al. 2023 ) , SGLang ( Zheng et al. 2023 ) , and TensorRT-LLM ( NVIDIA [n. d.] ) support compute-communication overlap for inference of large models such as Llama-3.3-70B that are sharded across GPUs in a tensor-parallel manner. In this paper, we present TokenWeave , an efficient method to minimize communication overhead for LLM inference. To the best of our knowledge, TokenWeave is the first system that can significantly reduce communication overheads even in batches with moderate number of tokens. For example, as shown in Figure 2 with tokens\_in\_batch=1024 , TokenWeave achieves latency improvements of 18\% on Llama-3.3-70B on 8xH100. On the other hand, state-of-the-art communication fusion methods like TileLink ( Zheng et al. 2025 ) , for the same setting, results in an overall hit in performance of 50+% over the non-overlapped baseline (Figure 14 ). Reducing the tokens\_in\_batch requirement makes TokenWeave a viable technique to use together with modern schedulers such as Sarathi ( Agrawal et al. 2024 ) which uses chunked prefill for efficient inference. For example, the popular vLLM ( Kwon et al. 2023 ) serving system uses a default chunked prefill size (i.e. max_num_batched_tokens) of 2048 ( vLLM [n. d.] ) . Even in disaggregated settings like Splitwise ( Patel et al. 2024 ) or DistServ ( Zhong et al. 2024 ) with tokens\_in\_batch\geq 4096 , Figure 2 shows that TokenWeave delivers over 20% gains and outperforms an equivalent model that has all communication removed. TokenWeave comprises of three key techniques. First, we propose Token-Splitting , which is a coarse-grained splitting of tokens in the input batch into just two approximately equal subsets so that the computation of one split can be overlapped with communication of the other. The token-based splitting (instead of request-based) allows us to split in either the batch- or sequence- dimensions independently. Naïvely splitting the input tokens into two equal halves can result in overheads due to wave-quantization. Thus, TokenWeave performs Smart-splitting , a GPU wave-aware splitting which ensures that the total number of waves executed in kernels of the two subtasks is not more than the number of waves in the full computation kernel. The coarse work partitioning and smart-splitting almost completely eliminates the compute overheads due to the smaller computations. Second, while state-of-the-art models today use the more efficient RMSNorm ( Zhang and Sennrich 2019 ) instead of LayerNorm for normalization, we find that even RMSNorm can take as much as 8% of the overall latency on H100s (Figure 3 ). Thus, unlike previous work such as Tilelink ( Zheng et al. 2025 ) , Flux ( Chang et al. 2024 ) and Nanoflow ( Zhu et al. 2024 ) that does not optimize normalization operations, TokenWeave carefully optimizes the normalization computation along with the communication operation. Figure 3 . RMSNorm overhead for different models vs sequence length on 8xH100 DGX (error bar shows std. dev. across 5 runs). RMSNorm performed after AllReduce