An Extensible Software Transport Layer for GPU Networking
Yang Zhou 1,2 Zhongjie Chen 3 Ziming Mao 1 ChonLam Lao 4 Shuo Yang 1 Pravein Govindan Kannan 5 Jiaqi Gao 6 Yilong Zhao 1 Yongji Wu 1 Kaichao You 1,3 Fengyuan Ren 3 Zhiying Xu 7 Costin Raiciu 8 Ion Stoica 1 1 UC Berkeley 2 UC Davis 3 Tsinghua University 4 Harvard University 5 IBM Research 6 Unaffilia · 2025
Read An Extensible Software Transport Layer for GPU Networking by Yang Zhou 1,2 Zhongjie Chen 3 Ziming Mao 1 ChonLam Lao 4 Shuo Yang 1 Pravein Govindan Kannan 5 Jiaqi Gao 6 Yilong Zhao 1 Yongji Wu 1 Kaichao You 1,3 Fengyuan Ren 3 Zhiying Xu 7 Costin Raiciu 8 Ion Stoica 1 1 UC Berkeley 2 UC Davis 3 Tsinghua University 4 Harvard University 5 IBM Research 6 Unaffilia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17307.txt.
Introduction
Yang Zhou and a large team of co-authors build a software layer that speeds up networking between graphics processors. Their paper is from two thousand twenty-five. Machine learning jobs spread across many such chips must swap data fast. The network cards in use rely on fixed hardware rules, which are hard to update as needs change.
Their layer, called UCCL, moves the control logic of the network into software on normal processors. The data path still runs in hardware at full speed. Software control makes new tricks possible, such as sending data along several paths at once. That avoids clashes where many flows fight for one path. In tests, shared learning tasks ran up to four and a half times faster than with standard cards.
Catalog record
| CREATOR | Yang Zhou 1,2 Zhongjie Chen 3 Ziming Mao 1 ChonLam Lao 4 Shuo Yang 1 Pravein Govindan Kannan 5 Jiaqi Gao 6 Yilong Zhao 1 Yongji Wu 1 Kaichao You 1,3 Fengyuan Ren 3 Zhiying Xu 7 Costin Raiciu 8 Ion Stoica 1 1 UC Berkeley 2 UC Davis 3 Tsinghua University 4 Harvard University 5 IBM Research 6 Unaffilia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Extensible Software Transport Layer for GPU Networking Yang Zhou 1,2 Zhongjie Chen 3 Ziming Mao 1 ChonLam Lao 4 Shuo Yang 1 Pravein Govindan Kannan 5 Jiaqi Gao 6 Yilong Zhao 1 Yongji Wu 1 Kaichao You 1,3 Fengyuan Ren 3 Zhiying Xu 7 Costin Raiciu 8 Ion Stoica 1 1 UC Berkeley 2 UC Davis 3 Tsinghua University 4 Harvard University 5 IBM Research 6 Unaffiliated 7 Amazon Web Services 8 University Pollitehnica of Bucharest & Broadcom Abstract . Fast-evolving machine learning (ML) workloads have increasing requirements for networking. However, host network transport on RDMA NICs is hard to evolve, causing problems for ML workloads. For example, single-path RDMA traffic is prone to flow collisions that severely degrade collective communication performance. We present UCCL, an extensible software transport layer to evolve GPU networking. UCCL decouples the data path and control path of existing RDMA NICs and efficiently runs the control-path transport on host CPUs. This software extensibility brings in transport innovations that cannot be achieved in hardware for ML workloads, e.g., a multipath transport to resolve flow collisions. ML collectives atop UCCL achieve up to 4.5 × \times higher performance compared to existing RDMA NICs. 1. Introduction Machine learning (ML) workloads and their requirements for networking are evolving rapidly. Less than ten years ago, deep neural networks only had millions of parameters, and were trained atop hundreds of CPUs/GPUs with parameter servers or allreduce collective communication ( Li et al., 2014 ) . After five years, large language models (LLMs) began to surge with billions of parameters, and were trained atop thousands of more powerful GPUs with multi-level parallelism and diverse collectives like allreduce, allgather, and reduce-scatter ( Grattafiori et al., 2024 ; Barham et al., 2022 ) . In the recent two years, large-scale LLM serving has become the norm; prefill-decode disaggregation ( Zhong et al., 2024 ) , as an efficient serving technique, requires fast peer-to-peer communication. This year, serving Mixture-of-Experts (MoE) models like DeepSeek-V3 ( Liu et al., 2024 ) became very popular, featuring challenging all-to-all communication among hundreds of GPUs. However, networking techniques especially the host network transport on RDMA NICs are hard to adapt and evolve to better suit the needs of ML workloads. Essentially, hardware changes are time-consuming and take much longer time than software changes. This can lead to a mismatch between the application needs and existing hardware optimizations, which often translates into poor performance. For example, Meta has reported that DCQCN ( Zhu et al., 2015 ) —a popular congestion control (CC) algorithm in datacenters supported by RDMA NICs—does not work well for LLM training workloads with low flow entropy and high traffic burstiness ( Gangidi et al., 2024 ) . As a result, Meta decided to disable the CC support in NICs and instead implement traffic scheduling at the application layer. Similarly, DeepSeek disabled the CC when running large-scale all-to-all for serving MoE models ( DeepSeek, 2025 ) . However, running a large-scale RDMA network without CC is brittle, as it can lead to deadlocks, head-of-line blocking, and pervasive congestion ( Hu et al., 2016 ; Guo et al., 2016 ; Mittal et al., 2018 ; Zhu et al., 2015 ; Bai et al., 2023 ; Gao et al., 2021 ) . In another example, Alibaba has observed severe performance degradation for collective communication during LLM training. This was due to the high level of flow collisions, which in turn was caused by the RDMA NICs supporting only single-flow/path per connection ( Qian et al., 2024 ) . To avoid this problem, Alibaba has redesigned the network topology for LLM training using a rail-optimized dual-plane architecture. However, such a redesign is costly to build and maintain. As we will show in § 3.2 , a software-only solution implementing multipath at the application level would have avoided such topology changes. In § 2.2 , we discuss four more examples where ML workloads require adapting RDMA hardware support, including network incast in MoE serving, semi-reliable gradient transmission, efficient loss recovery, and avoiding vendor lock-in. Furthermore, the existing hardware-baked host network transport layer makes it difficult to productize new research proposals ( Wang et al., 2024 ; Le et al., 2024 ; Bonato et al., 2024 ; Olteanu et al., 2022 ) to improve the performance of ML workloads in real-world scenarios. These examples point to network extensibility as one of the key challenges in datacenter networks. In this paper, we focus on addressing this challenge in the context of RDMA-enabled hosts, by proposing and implementing a software-only extensible transport layer called UCCL (Ultra-CCL). This software-only approach makes it much easier to efficiently support new ML workloads, e.g., CC algorithms that can better support ML traffic than existing DCQCN, and multipathing to mitigate flow collisions. To implement this approach, we need to address two challenges: (1) How to decouple the data and control paths for existing RDMA NICs? The data path handles network data transfers for the GPU, while the control path manages transport control decisions like CC, packet reliability, and multipath load balancing (LB). Only if the control path is decoupled from the data path can we implement it on the CPU. (2) How can we achieve hardware-level performance with the control path running on the CPU? This is challenging given the inter-GPU-server high bandwidth, which can exceed 3.2 Tbps ( Amazon Web Services, 2025 ; Microsoft Azure, 2024 ) . To address the first challenge, UCCL repurposes the features of existing RDMA NICs to achieve efficient data and control path separation. In particular, UCCL leverages RDMA Unreliable Connection (UC) to bypass hardware-baked CC and packet reliability logic, and uses RDMA immediate data to pass transport control states between sender and receiver CPUs. For some NICs that do not support UC, such as AWS EFA ( Services, 2025 ) , UCCL leverages the scatter-gather feature over RDMA Unreliable Datagram (UD) to achieve separation. To address the second challenge, UCCL leverages an array of techniques tailored to the unique characteristics of ML workloads. On the one hand, UCCL leverages GPUDirect ( Corporation, 2025f ) to alleviate CPU overhead. On the other hand, UCCL employs control coalescing to make transport control decisions for every 32KB data chunk rather than every packet; this works well as many transport decisions such as CC do not require per-packet reaction, but just per-RTT to avoid overreaction ( Li et al., 2019 ; Google, 2024 ) . As a result, UCCL can handle 400 Gbps unidirectional traffic with a single CPU core, and achieve the same message latency as hardware-based transport. UCCL leverages 256 RDMA QPs (Queue Pairs) per connection to perform multipathing, without worrying about the high QP context swapping overhead highlighted in previous work ( Wang et al., 2023 ) ; this is because ML workloads feature bulk data transfer with mostly MTU-sized large packets, which effectively amortizes the swapping overhead. UCCL’s software transport is practical and economical in modern GPU servers, which usually have hundreds of powerful CPU cores; in fact, these CPU cores are often heavily underutilized. For example, from our private conversation with the model training team of a major GPU vendor, their CPU utilization is averagely 14.5% out of 128 cores, when