GICC: A High-Performance Runtime for G PU- I nitiated C ommunication and C oordination in Modern HPC Systems Conference: The 35th International Symposium on High-Performance Parallel and Distributed Computing; July 13–16, 2026; Cleveland, OH, USA The 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’26), July 13–16, 2026, Cleveland, OH, USA DOI: 10.1145/3806645.3807576 ISBN: 979-8-4007-2640-8/2026/07 CCS: Computer systems organization Distributed architectures CCS: Networks
Baodi Shan, Mauricio Araya-Polo, Barbara Chapman · 2026
Read GICC: A High-Performance Runtime for G PU- I nitiated C ommunication and C oordination in Modern HPC Systems Conference: The 35th International Symposium on High-Performance Parallel and Distributed Computing; July 13–16, 2026; Cleveland, OH, USA The 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’26), July 13–16, 2026, Cleveland, OH, USA DOI: 10.1145/3806645.3807576 ISBN: 979-8-4007-2640-8/2026/07 CCS: Computer systems organization Distributed architectures CCS: Networks by Baodi Shan, Mauricio Araya-Polo, Barbara Chapman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22126.txt.
Introduction
Shan, Araya-Polo, and Chapman present GICC, a runtime letting GPU kernels trigger network-card work directly, host-free on the fast path. On OFI interconnects like HPE Slingshot, current runtimes need host-driven progress and lack a bounded way to recycle pre-staged NIC work. GICC splits coordination meaning from data movement and adds asynchronous resource recycling, with the network card signalling completion to GPU and host memory. Built on NVIDIA and AMD GPUs over InfiniBand and Slingshot, GICC sustains GPU coordination under finite network state.
Catalog record
| CREATOR | Baodi Shan, Mauricio Araya-Polo, Barbara Chapman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65628 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22126
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GICC: A High-Performance Runtime for G PU- I nitiated C ommunication and C oordination in Modern HPC Systems Conference: The 35th International Symposium on High-Performance Parallel and Distributed Computing; July 13–16, 2026; Cleveland, OH, USA The 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’26), July 13–16, 2026, Cleveland, OH, USA DOI: 10.1145/3806645.3807576 ISBN: 979-8-4007-2640-8/2026/07 CCS: Computer systems organization Distributed architectures CCS: Networks Baodi Shan, Mauricio Araya-Polo, Barbara Chapman Abstract . An increasingly popular approach for distributed GPU applications is kernel-level, cross-node coordination to reduce launch overheads and improve compute–communication overlap. The reality is that such support is lacking. On one hand, on OFI-based interconnects such as HPE Slingshot—which powers six of the top ten systems in the November 2025 Top500, including the top three—GPU kernels cannot autonomously drive distributed coordination: existing runtimes rely on host-driven progress and do not provide a bounded mechanism for recycling pre-staged NIC work across repeated GPU-triggered operations. On the other hand, on InfiniBand, GPU-initiated communication is possible, but current implementations incur unnecessary synchronization and locking overheads. This paper presents GICC, a GPU-driven coordination framework that enables GPU kernels to issue coordination commands and directly trigger NIC-level operations without host involvement on the fast path. For example, in stencil computations, GPU threads can directly initiate halo exchanges with neighboring nodes as soon as boundary regions are computed, rather than waiting for the kernel to complete and relying on the host to orchestrate communication. This eliminates synchronization overhead and enables fine-grained overlap between interior computation and boundary data transfer. GICC decouples coordination semantics from data movement and introduces an asynchronous resource reclamation scheme: the NIC signals completion to both GPU and host memory, enabling a lightweight host thread to recycle NIC resources concurrently with GPU execution—without injecting latency into the coordination path. This design enables sustained GPU-driven coordination under finite NIC state, a capability absent from existing runtimes on OFI-based fabrics. We implement GICC on NVIDIA and AMD GPUs over both InfiniBand and Slingshot (OFI). On Slingshot, GICC reduces average per-coordination latency by up to 229 × 229\times and improves weak scaling efficiency by up to 25%. On InfiniBand, GICC achieves up to 1.95 × 1.95\times lower put latency than NVSHMEM by eliminating unnecessary locking and synchronization. For an industrial stencil-based proxy application on 64 AMD MI250X GCDs, GPU-aware MPI incurs over 52% higher communication time than GICC, and GICC achieves 42% parallel efficiency versus MPI’s 35.4%. These results demonstrate that GPU-driven, resource-aware coordination is both necessary and achievable across the dominant interconnects in modern HPC systems. † † cc-license: by 1. Introduction Modern large-scale high-performance computing (HPC) applications increasingly rely on GPUs to deliver extreme computational throughput. In many distributed GPU workloads, execution proceeds as a sequence of short GPU phases—e.g., pipelined stages, fused kernels, or iterative loops—interleaved with coordination points such as barriers, reductions, and phase ordering. Under today’s programming and runtime models, these coordination points are typically host-driven: GPU kernels must return control to the CPU, which invokes a runtime to orchestrate collective operations, enforce ordering, and manage progress. Even when the coordination payload is small, this host round-trip fragments GPU execution, introduces repeated kernel launches and host–device synchronization, and limits the extent to which GPUs can govern distributed control flow. As shown in Section 3 , a 200-phase workload can spend over 32% of its runtime in coordination overhead alone. GICC addresses two complementary gaps in existing GPU communication frameworks. The first is GPU-triggered communication on OFI-based fabrics such as HPE Slingshot. Existing runtimes on Slingshot including NVSHMEM ( 17 ) are host-mediated—GPU threads may invoke GPU-visible APIs, but the host CPU still mediates work submission and progress. GICC enables GPU-triggered data movement on these fabrics, in which GPU kernels can trigger pre-configured NIC operations without a CPU round trip on the fast path. The second is GPU-driven coordination . Data movement alone is insufficient for distributed GPU applications: while put/get-style operations move bytes, coordination defines ordering and synchronization semantics (e.g., barriers and phase transitions) that must make forward progress under system constraints. We use GPU-driven coordination to mean that GPU control flow governs when coordination occurs and can observe completion conditions within a kernel. On OFI-based systems, where device code cannot enqueue new network work descriptors, this requires a GPU-triggered execution style in which the host pre-configures bounded NIC work and the GPU triggers execution on the fast path. GICC introduces GPU-driven coordination primitives—including active messages and barriers—that allow GPU control flow to govern distributed synchronization across both InfiniBand and OFI/CXI fabrics. Figure 1 summarizes GICC’s two-layer design. Block diagram of GICC with two fabric-specific paths. The InfiniBand path shows a GPU kernel and an InfiniBand NIC with direct interaction. The OFI/CXI path shows a GPU kernel and a Slingshot/CXI NIC, plus a host monitor thread that runs concurrently and interacts with both the GPU and NIC to provide progress and manage NIC state. Figure 1 . Overview of GICC. On InfiniBand, GPUs directly interact with the NIC on the fast path. On OFI/CXI, GPUs drive coordination within kernels while a lightweight host monitor runs concurrently to provide progress and manage NIC state. Block diagram of GICC with two fabric-specific paths. The InfiniBand path shows a GPU kernel and an InfiniBand NIC with direct interaction. The OFI/CXI path shows a GPU kernel and a Slingshot/CXI NIC, plus a host monitor thread that runs concurrently and interacts with both the GPU and NIC to provide progress and manage NIC state. Realizing GPU-driven coordination is fundamentally non-trivial. GPU kernels cannot block or issue system calls, lack a general-purpose progress engine, and execute under a SIMD-style model that complicates fine-grained control flow. On the dominant OFI-based interconnects accessed via libfabric ( 20 ) , progress and resource retirement may be mediated by host-side software (e.g., manual provider progress), and modern NICs expose limited on-device state for triggered work. These constraints mean that naive approaches (e.g., simply polling GPU-visible flags or pre-staging large numbers of operations) do not provide sustained, correct coordination under realistic NIC resource limits. This is a significant shortcoming given the prevalence of such systems in modern HPC. For instance, among the ten highest-ranked systems in the November 2025 Top500 ( 28 ) , six deploy HPE Slingshot as their primary interconnect, including El Capitan, Frontier, and Aurora. Yet existing GPU communication frameworks do not provide a portable GPU-driven coordination model on these OFI-based fabrics. Vendor runtimes have begun exposing GPU-triggered interfaces, but these capabilities are limited in scope: they focus on point-to-point data movement and do not provide device-side collective coordination or kernel-level integration of distributed synchronization with GPU control flow. Consequently, GPUs on today’s largest HPC systems still lack the ability to autonomously drive distributed coordination ( 29 ) . This paper presents GICC , a GPU-driven distributed coordination framework designed explicitly for OFI-based interconnects such as HPE Slingshot, while also providing a streamlined path on GPU-native fabrics such as InfiniBand. Our primary systems contribution targets the OFI/CXI setting, where the coordination gap is most acute; on InfiniBand, GICC offers a lighter device-side path that exposes the same coordination interface but benefits from richer GPU–NIC integration. Even on GPU-native fabrics, existing GPU communication runtimes can incur avoidable device-side