CUCo: An Agentic Framework for Compute and Communication Co-design Conference: ; ;
Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, Aditya Akella · 2026
Read CUCo: An Agentic Framework for Compute and Communication Co-design Conference: ; ; by Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, Aditya Akella (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02376.txt.
Introduction
Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, and Aditya Akella automate GPU kernel writing in this 2026 paper. Kernels are the small programs that run the math on graphics chips. In large model training, kernels must move data between chips as well as compute. Hand-writing kernels that do both well is slow and error-prone.
Their framework, CUCo, uses AI agents and needs no training of its own. Agents generate kernels that orchestrate compute and communication together. That joint view opens gains that compute-only tuning cannot reach. Prior tools focused almost only on computation, leaving communication largely untouched. CUCo beat leading baselines and cut end-to-end latency by up to about one and a half times. The work targets large-scale training and inference alike.
Catalog record
| CREATOR | Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, Aditya Akella |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66140 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CUCo: An Agentic Framework for Compute and Communication Co-design Conference: ; ; Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, Aditya Akella Abstract . Custom CUDA kernel development is essential for maximizing GPU utilization in large-scale distributed LLM training and inference, yet manually writing kernels that jointly leverage both computation and communication remains a labor-intensive and error-prone process. Prior work on kernel optimization has focused almost exclusively on computation, leaving communication kernels largely untouched even though they constitute a significant share of total execution time. We introduce CUCo , a training-free agent-driven workflow that automatically generates high-performance CUDA kernels that jointly orchestrate computation and communication. By co-optimizing these traditionally disjoint components, CUCo unlocks new optimization opportunities unavailable to existing approaches, outperforming state-of-the-art baselines and reducing end-to-end latency by up to 1.57 × \times . 1. Introduction The rapid scaling of machine learning models has made distributed training and inference across large GPU clusters the de facto standard. Meeting the performance demands of this paradigm requires efficiency at every level: both computation and communication kernels have been heavily optimized, and the overlap between the two has emerged as a critical axis for further improvement. Achieving this overlap requires pipelining communication with computation, hiding network latency behind useful GPU work. Historically, however, programming models and vendor-specific constraints prevented GPU compute kernels from directly invoking communication primitives ( 14 ; 3 ) , forcing host-side orchestration in which the CPU manages the handoff between computation and asynchronous communication. This indirection introduces synchronization overhead and limits how tightly the two can be interleaved. As a result, compute and communication libraries such as cuBLAS/, cuDNN ( 5 ; 22 ) and NCCL ( 24 ) have evolved largely in isolation. Recent advances, however, have begun to bridge this barrier. Interfaces such as NVSHMEM ( 25 ) and NCCL’s device-side ( 23 ) APIs now allow GPU kernels to invoke communication primitives directly, enabling distributed logic to be expressed within a single kernel or across tightly coordinated kernels in concurrent streams. A distributed matrix multiplication can now be performed entirely on the GPU without host intervention. This represents a fundamental shift in how kernels are designed for distributed machine learning: rather than treating computation and communication as separate phases orchestrated by the host, they can now be co-designed or even fused within the GPU itself. Emerging libraries ( 41 ; 10 ; 3 ; 33 ) exemplify this trend, demonstrating both the performance potential of tightly integrating compute and communication and the significant architectural complexity such integration introduces. Yet, they also reveal a challenge: kernels that jointly manage computation and communication are extremely difficult to write well. First, designing high-performance CUDA kernels that jointly orchestrate computation and communication requires navigating a complex and enormous design space; effective strategies are counterintuitive, and existing tooling provides limited guidance. The relevant decisions—where to initiate communication, when to synchronize, how to define tiling boundaries, and how to size chunks ( 3 ; 44 ) —interact in non-trivial ways, and the right configuration is highly sensitive to the specific operation and hardware context. Even seemingly local choices have global consequences: in distributed matrix multiplication, triggering communication at tile granularity maximizes theoretical overlap but can introduce enough synchronization overhead to erase any benefit ( 10 ) , yet the optimal granularity cannot be derived from first principles and must be discovered empirically ( 45 ) . Modern ML compilers offer no relief—they assume communication lives outside the kernel boundary ( 44 ; 34 ; 11 ) , treating it as a host-side side effect, and moving it inside violates the assumptions underlying operator scheduling and tiling, breaking memory access patterns in intricate ways neither the compute compiler nor the communication runtime was designed to resolve ( 27 ) . Secondly, even if an optimal execution plan is identified for a specific configuration, the design space complexity is exacerbated by hardware heterogeneity. While GPU architectures offer a discrete set of design points, the underlying network fabrics exhibit high variance in bandwidth, latency, and topology ( 39 ; 29 ; 9 ) (e.g. InfiniBand vs. RoCE, Fat-Tree vs. Torus). This variability creates a search space where the optimal kernel configuration is tightly coupled with both the device type and the specific interconnect characteristics ( 3 ; 41 ) . Manually tuning kernels for every possible combination of compute and network architecture is intractable ( 4 ; 42 ) . Consequently, the transition to device-side communication necessitates a move away from manual kernel engineering toward automated, hardware-aware optimization frameworks ( 42 ; 44 ; 43 ) . To overcome these challenges, we argue for an automated ground-up agentic framework with three inter-twined components: (i) a structured design space specification that grounds agent reasoning in communication design decisions, API semantics, and hardware context; (ii) a fast-path agent that prioritizes correctness, producing workable kernels without getting trapped in long generation cycles; and (iii) a slow - path agent that leverages empirical feedback to reason about subtle compute–communication interference patterns, ultimately converging on high - performance implementations tailored to each workload and hardware topology. Each component is essential. Without a clear declarative interface, the framework cannot ground its reasoning and risks hallucinating invalid co-design strategies. Without the fast - path agent, it would over - optimize prematurely, breaking correctness and failing to recover. Without the slow - path agent, it would miss rich optimization opportunities, leaving substantial performance gains unrealized. We present CUCo , an agentic framework that introduces a minimal, declarative set of communication primitives that transforms compute–communication co-design into a structured and intentionally constrained design space. Within this space, co-design becomes a search problem over only the schedules that these primitives can express, which sharply narrows the agent’s degrees of freedom while preserving meaningful flexibility. This structure grounds the agent’s synthesis in valid collective semantics, reducing invalid or hallucinated optimizations while leaving ample room to explore diverse scheduling strategies and overlap patterns. CUCo ’s fast-path agent leverages the communication primitives to rapidly prototype compute-communication kernels. It prioritizes correctness over performance: rather than aggressively interleaving computation and collectives, it composes them into a single kernel with explicit, barrier-delimited phases. These barriers enforce a coarse-grained global schedule in which all ranks advance through the same sequence of compute and communication steps in lockstep. This prevents the distributed divergence that can deadlock collectives or leave cross - device data in an inconsistent state, making verification and debugging painful. This produces a reliable, easy-to-validate baseline that the slow-path agent can later relax and refine. CUCo ’s slow - path agent complements the fast - path agent with an evolution - style optimization loop that prioritizes diversity and performance. Starting from the fast - path’s distributed - safe baseline, the slow path explores the design space by mutating how collectives are chunked, how synchronization boundaries are positioned, and how compute is interleaved with communication. It then compiles and runs these candidates, using empirical feedback (e.g. latency, throughput, and achieved bandwidth/SM occupancy) to guide the search, eliminating underperforming designs while refining the promising ones. Through repeated iterations, the slow path converges on high - performance compute - communication kernels tailored to the workload and the underlying hardware topology. CUCo is implemented in