Prime Collective Communications Library - Technical Report
Michael Keiblinger, Mario Sieg, Jack Min Ong, Sami Jaghouar, Johannes Hagemann · 2025
Read Prime Collective Communications Library - Technical Report by Michael Keiblinger, Mario Sieg, Jack Min Ong, Sami Jaghouar, Johannes Hagemann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14065.txt.
Introduction
Michael Keiblinger and co-authors present a communications library for distributed machine learning in a 2025 technical report. Training across the public internet faces peers that join, leave, or fail mid-run. Standard libraries assume a fixed, reliable group.
Their library, PCCL, lets peers join dynamically and recover from failure while collective steps like all-reduce keep running. It spreads traffic across many connections to use long cross-continent links well, reaching high bandwidth in tests. Stress tests across major operating systems passed under rapid churn. The work targets training setups that cannot rely on a private data center.
Catalog record
| CREATOR | Michael Keiblinger, Mario Sieg, Jack Min Ong, Sami Jaghouar, Johannes Hagemann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14065
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prime Collective Communications Library - Technical Report Michael Keiblinger, Mario Sieg, Jack Min Ong, Sami Jaghouar, Johannes Hagemann Abstract This report presents the Prime Collective Communications Library (PCCL), a novel fault-tolerant collective communication library designed specifically for distributed machine learning workloads over the public internet. PCCL introduces a new programming model that enables dynamic peer joining and failure recovery. The library implements efficient collective operations like all-reduce while providing robust fault tolerance mechanisms that allow the system to continue operating even when peers fail or join during ongoing operations. We demonstrate that PCCL’s design enables practical solutions to dynamic membership challenges in workloads with repeated operations and deterministic state advancement. Our implementation passes extensive stress tests across all major operating systems, showing reliable operation even under rapid peer churn and concurrent collective operations. By dispatching to multiple connections, we can efficiently utilize cross-continental long-fat-pipe TCP WAN links, in our experiments achieving up to 45 Gbit/s of bandwidth utilization across western Europe and 25 Gbit/s across North America and Europe. PCCL’s architecture enables easy implementation of distributed low-communication optimization strategies like DiLoCo, which significantly reduce communication frequency. Combined with quantization, this leads to a significant reduction in the bandwidth required for distributed training workloads. PCCL also allows for concurrent collective operations, which enables optimization strategies like async DiLoCo, which can completely hide communication overhead by implementing one-step delayed parameter updates. PCCL can facilitate exact bit-parity of the shared state across peers in all cases induced by graceful or abrupt peer churn. While PCCL exposes a C99 API, Python bindings are available which are compatible with PyTorch alongside FSDP. PCCL is available under the open source MIT license. 1 Introduction Over the past decade, the increasing scale and complexity of machine learning (ML) models have driven the adoption of large-scale distributed training across clusters of GPUs or other specialized accelerators. These workloads are especially communication-intensive, as parameter updates and gradient calculations must be synchronized across multiple worker nodes. To handle this synchronization, collective communication libraries like MPI and, more recently, NVIDIA Collective Communications Library (NCCL) ( NVIDIA Corporation, 2025 ) have played a central role in providing efficient all-reduce, broadcast, and other collective operations on tightly coupled, high-performance clusters. Despite their success, these traditional libraries were not designed to handle the realities of modern, highly dynamic environments, where resources may join or leave frequently and where connectivity spans public clouds or other wide-area networks. They often assume fixed membership, predictable node connectivity, and uniform performance characteristics—assumptions that rarely hold at large scale or outside dedicated HPC clusters. As a result, when failures or dynamic reconfigurations occur, these libraries do not gracefully handle peer churn, and thus prevent resilient training. In this work, we introduce the Prime Collective Communications Library ( PCCL ), a novel fault-tolerant collective communication library that addresses these fundamental limitations. PCCL is designed specifically for distributed ML workloads operating over public networks, where peers may fail, join, or disconnect at any point during training. Our approach enables dynamic peer membership and incorporates robust fault-tolerance mechanisms that allow the system to continue collective operations even as peers come and go. By exploiting the deterministic nature of most optimizers, where bit-identical state progression is guaranteed if all peers receive identical updates - even when performed on GPUs, PCCL ensures consistent model synchronization across nodes without additional communication beyond the collective operation results. Unlike traditional HPC libraries, PCCL leverages a master-client model that accommodates dynamic membership and continuous topology optimization. New peers join as "registered," then graduate to full participation through a quick consensus phase. Each collective operation proceeds via carefully orchestrated micro-consensus steps—ensuring fault tolerance and consistent model states, even if peers leave mid-operation. In the following sections, we detail how these design decisions enable smooth peer churn, p2p bandwidth measurements for ring-based optimization, and bit-parity among peers inside the shared state without unnecessary re-transmissions. 2 PCCL Architecture 2.1 Overview PCCL introduces a novel approach to collective communications for distributed machine learning that differs significantly from classic HPC libraries such as OpenMPI or NCCL. This section describes the design rationale behind PCCL’s master-client model, its fine-grained state machine, and the mechanisms it employs to ensure fault tolerance and dynamic peer membership. The following subsections detail each component, providing insights into how PCCL handles joining or failing peers mid-run, optimizes its ring topology based on bandwidth measurements, and enforces bit-identical shared state across participants. 2.2 Master–Client Model Unlike traditional libraries that assume all processes start simultaneously, PCCL leverages a lightweight master process that orchestrates membership and consensus of collective operations. • Clients (Peers) : Each training process instantiates a PCCL communicator and connects to the master. Once a peer is "registered," it can be voted into the active group. • Master Node : Maintains a view of which peers are accepted, tracks the ring topology (for all-reduce operations), and coordinates all state transitions through micro-consensus steps. 2.2.1 Membership Lifecycle When a new peer connects, it enters an initial registered phase. Accepted peers periodically vote on whether to admit newly registered peers into their group. If they do so, the master instructs everyone to establish direct p2p connections as needed. This approach allows peer churn without forcing restarts of the entire training job. 2.2.2 What the Master Tracks Clients and Their Status. The master differentiates between registered peers (not yet active) and accepted peers (fully integrated into the collective). Ring Topology and Bandwidth Metrics. Based on p2p bandwidth tests, the master attempts to optimize the ring order, solving an asymmetric traveling-salesman problem ( Jaghouar et al., 2024 ) . For this purpose, we have developed a custom heuristic and exact ATSP solver called libtsp ( Keiblinger, 2025 ) . PCCL’s optimization can occur as a quick pass (immediate solution within a short timeout) or a more involved "moonshot" background computation. Once a new or improved ring order is established, the master instructs peers to reconfigure their p2p connections accordingly. Shared State. The master ensures that each peer has the same shared state. Any peer with an out-of-date revision or mismatched content is instructed to request data from an up-to-date peer. This ensures that all participants converge on a bit-identical set of parameters and optimizer states. Collective Operations. The master coordinates all active collective operations, preventing conflicts between accepting new peers, synchronizing shared state, and performing ring-based collectives. While it does coordinate the collective operations, it is not involved in the actual collective operations nor shared state re-transmissions themselves. It merely ensures consensus of all peers that the collective operation has completed and propagates IO failures to potentially interrupt ongoing collective operations to avoid stall. 2.3 Collective Operations and States PCCL manages each operation (e.g., an all-reduce) through well-defined states. This state machine approach enables graceful error handling if a peer drops out mid-operation and ensures that every peer transitions in lockstep. 2.3.1 Phases and State Machine Each peer moves through a sequence of phases: • ConnectionPhase : PEER_REGISTERED or PEER_ACCEPTED . • ConnectionState : e.g. IDLE , VOTE_ACCEPT_NEW_PEERS , COLLECTIVE_COMMUNICATIONS_RUNNING , etc. • CollectiveCommunicationState (per tag) : VOTE_INITIATE_COLLECTIVE_COMMS , PERFORM_COLLECTIVE_COMMS , VOTE_COMPLETE_COLLECTIVE_COMMS . Each transition requires unanimous agreement among involved peers ("micro-consensus") so that inconsistencies or partial failures can be detected and handled