FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
Qingyuan Li, Bo Zhang, Hui Kang, Tianhao Xu, Yulei Qian, Yuchen Xie, Lin Ma · 2025
Read FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication by Qingyuan Li, Bo Zhang, Hui Kang, Tianhao Xu, Yulei Qian, Yuchen Xie, Lin Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03760.txt.
Introduction
Qingyuan Li and co-authors speed up chatter between graphics chips that train large language models in this 2025 paper. As models grow, moving data between chips becomes the bottleneck. Sending data in fewer bits helps, but chips only accept set bit widths, and squeezing too hard wrecks accuracy.
Their system, FlashCommunication V2, splits odd bit widths into basic units that any hardware can carry. It also sets aside extreme values and sends them at full precision, which keeps quality loss acceptable even at two bits per value. Careful joint design of software and hardware gave speed gains of several times on common link types.
Catalog record
| CREATOR | Qingyuan Li, Bo Zhang, Hui Kang, Tianhao Xu, Yulei Qian, Yuchen Xie, Lin Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03760
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication Qingyuan Li, Bo Zhang, Hui Kang, Tianhao Xu, Yulei Qian, Yuchen Xie, Lin Ma Abstract Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2 × \times speedup in AllReduce and 2 × \times in All2All communication. Introduction In the fast-evolving field of AI, large language models (LLMs) favor mixture-of-experts (MoE) for strong performance with efficient training and inference. DeepSeek-V3 ( DeepSeek-AI 2025 ) , a 671B-parameter mixture-of-experts (MoE) model with 37B activated parameters per token, requires only 2.788M H800 GPU hours for training. MiniMax-01 ( MiniMax 2025 ) is a model with 32 experts and a total of 456 billion parameters (45.9 billion parameters activated) equipped with lightning attention, which allows a context window of several million tokens. Qwen3 ( Qwen 2025 ) ’s flagship Qwen3-235B-A22B is also a competitive MoE model in coding, math, and general tasks, featuring hybrid thinking modes and support for 119 languages. Kimi K2 ( Kimi 2025 ) , a 1T-parameter MoE model (32B activated), focuses on frontier knowledge and agentic capabilities. Together, these models drive progress in performance, efficiency, and versatility, shaping AI’s future across various domains. Figure 1: Llama-3-8B’s C4 perplexity across common bit widths using various quantization schemes. To deploy trillion-parameter MoE LLMs in practice, it requires meticulous hardware-software codesign integrating multiple levels of parallelism, highly-optimized scheduling policies, and efficient kernel implementation. DeepSeek-V3 addresses hardware bottlenecks through innovations like Multi-head Latent Attention (MLA) for memory efficiency, Mixture of Experts (MoE) for computation-communication trade-offs, FP8 mixed-precision training, and a Multi-Plane Network Topology, while highlighting future hardware directions such as low-precision units and low-latency communication. Huawei CloudMatrix384 ( Zuo et al. 2025 ) presents a next-gen AI datacenter architecture with 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via a high-bandwidth Unified Bus, paired with CloudMatrix-Infer for peer-to-peer serving, large-scale expert parallelism, and hardware-aware optimizations (e.g., INT8 quantization) to balance throughput and latency. Step-3 ( StepFun 2025 ) achieves cost-effective decoding through Multi-Matrix Factorization Attention (MFA) to reduce KV cache and computation, and Attention-FFN Disaggregation (AFD) for distributed inference, outperforming DeepSeek-V3 in throughput. DeepEP ( Zhao et al. 2025b ; Team and Contributors 2025 ) , a communication library for MoE and expert parallelism, offers high-throughput/low-latency kernels (supporting FP8) optimized for prefill (normal dispatch) and decode (low-latency dispatch) phases, with features like SM-free communication-computation overlap. Prefill-decoding disaggregation has also become a common practice to optimize LLM serving performance and resource use. DistServe ( Zhong et al. 2024 ) optimizes LLM serving by disaggregating prefill and decoding across GPUs, co-optimizing resource allocation and parallelism to boost throughput under latency constraints. Mooncake ( Qin et al. 2024 ) employs a KVCache-centric disaggregated setup, separating prefill/decoding clusters, using underutilized GPU resources for KVCache, and adding a scheduler with overload rejection to enhance long-context throughput. MegaScale-Infer ( Zhu et al. 2025 ) addresses MoE inference via disaggregating attention and FFN modules, enabling independent scaling, with ping-pong pipelining and a high-performance communication library to maximize GPU utilization and throughput. Finally, the SGLang team ( Team and Contributors 2025 ) deploys DeepSeek with Prefill-Decode (PD) disaggregation and large-scale expert parallelism on 96 H100 GPUs leverages DeepEP, DeepGEMM, Two-batch Overlap (TBO), and Expert Parallelism Load Balancer (EPLB) to boost throughput, reduce costs, and balance workloads across GPUs. To resolve the widespread communication bottleneck introduced by distributed training (e.g., AllReduce in tensor parallelism, All2All communication in expert parallelism) and deployment (e.g., KV-cache communication in Prefill-decoding disaggregation), we introduce a novel communication optimization method called Flash Communication V2. Specifically, we compress the communication volume with quantization at any possible bit width, as shown in Figure 1 . To achieve a practical performance boost, we employ bit-splitting , which reorganizes the data storage mechanism to handle irregular bit widths. Also, we involve spike reserving to manage outliers in lower precision without incurring too much overhead. Our overall results are demonstrated by Llama-3-8B’s TTFT measurement on various modern GPUs in Figure 2 . Figure 2: Llama-3-8B’s TTFT across modern GPUs (TP=8) under various precision settings. Related Work Communication and Computation Overlap. Several works focus on optimizing communication and computation overlap in large-scale model training and inference. ScMoE ( Cai et al. 2025 ) addresses the All2All communication bottleneck by decoupling communication from conventional sequential ordering via a shortcut-connected MoE architecture, achieving 1.49 \times training and 1.82 \times inference speedups. Domino ( Wang et al. 2024 ) eliminates AllReduce communication overhead in LLM training by breaking batch data dependency into micro-batches and pipelining them, yielding up to 1.3 \times speedup over Megatron-LM ( NVIDIA 2024 ) . DeepSeek-V2 ( Liu et al. 2024a ) overlaps the computation of shared experts with the expert parallel All2All communication. DeepSeekV3 ( Zhao et al. 2025a ) overlaps All2All communication of MoEs over attention computation through micro-batches scheduling, also adopted by MiniMax-01 ( MiniMax 2025 ) . Communication Quantization. DeepSeekV3 quantizes activation into FP8 to have low-precision communication ( DeepSeek-AI 2025 ) . ZeRO++ ( Wang et al. 2023 ) addresses the communication volume limitations of the Zero Redundancy Optimizer (ZeRO) in giant model training, especially in low-bandwidth clusters or at scale with small per-GPU batch sizes. It introduces three techniques: block-quantization based AllGather, data remapping that trades communication for memory, and an All2All-based quantized gradient averaging paradigm replacing reduce-scatter, collectively reducing ZeRO’s communication volume by 4 \times and enabling up to 2.16 \times better throughput at 384 GPU scale. Flash Communication ( Li et al. 2024b ) , on the other hand, targets the tensor-parallelism communication bottleneck in large language model inference, particularly on devices with limited bandwidth, using an optimized fusion kernel called Flash-AllReduce to boost intra-node communication speed by over 3 \times and reduce time-to-first-token by 2 \times with nearly no loss in model accuracy, as demonstrated across various up-to-date LLMs. Low-bit Quantization. LLM quantization methods focus on reducing memory usage and boosting efficiency with minimal loss of accuracy, each with its unique approaches and trade-offs. AWQ ( Lin et al. 2024 ) protects 1% activation-identified salient weight channels via up-scaling, generalizing well but focusing on weight-only quantization. OmniQuant ( Shao et al. 2023 ) optimizes quantization hyperparameters via learnable weight clipping and learnable equivalent transformation, supporting diverse settings but demanding more computation. For extremely low-bit quantization, Norm Tweaking ( Li et al. 2024a ) updates only the weights of normalization layers to mitigate the quantization errors, excelling in low-bit