cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution

Jinwu Chen, Qidie Wu, Bin Li, Lin Ma, Xin Si, Yang Hu, Shouyi Yin, Jun Yang · 2025

Read cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution by Jinwu Chen, Qidie Wu, Bin Li, Lin Ma, Xin Si, Yang Hu, Shouyi Yin, Jun Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16465.txt.

Introduction

Jinwu Chen and seven co-authors automate GPU programming in a paper that appeared on arXiv in 2025. CUDA kernels, the small programs that run on graphics cards, decide the speed of AI workloads, but tuning them demands rare hardware expertise. Language models paired with evolutionary search can write kernels, yet weak agent designs hold them back.

Their framework, cuPilot, coordinates several agents around an intermediate representation called strategy, a semantic sketch of how a kernel should be optimized. A strategy-coordinated evolution algorithm, roofline-guided prompts that reason about hardware limits, and strategy-level seeding of the population are its main parts. On a benchmark of 100 kernels, cuPilot's output ran on average 3.09 times faster than PyTorch. On matrix multiplication tasks it found sophisticated optimizations with high hardware utilization. The kernels are open-sourced.

Catalog record

CREATORJinwu Chen, Qidie Wu, Bin Li, Lin Ma, Xin Si, Yang Hu, Shouyi Yin, Jun Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16465

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution Jinwu Chen, Qidie Wu, Bin Li, Lin Ma, Xin Si, Yang Hu, Shouyi Yin, Jun Yang Abstract . Optimizing CUDA kernels is a challenging and labor-intensive task, given the need for hardware-software co-design expertise and the proprietary nature of high-performance kernel libraries. While recent large language models (LLMs) combined with evolutionary algorithms show promise in automatic kernel optimization, existing approaches often fall short in performance due to their suboptimal agent designs and mismatched evolution representations. This work identifies these mismatches and proposes cuPilot, a strategy-coordinated multi-agent framework that introduces strategy as an intermediate semantic representation for kernel evolution. Key contributions include a strategy-coordinated evolution algorithm, roofline-guided prompting, and strategy-level population initialization. Experimental results show that the generated kernels by cuPilot achieve an average speed up of 3.09 × \times over PyTorch on a benchmark of 100 kernels. On the GEMM tasks, cuPilot showcases sophisticated optimizations and achieves high utilization of critical hardware units. The generated kernels are open-sourced at https://github.com/champloo2878/cuPilot-Kernels.git. 1. Introduction The optimization of CUDA kernels, a widely adopted GPU programming model ( 9 ) , is crucial for achieving peak AI inference/training performance. However, optimizing GPU kernels is a labor-intensive and time-consuming task. First, it necessitates a thorough understanding of the target GPU architecture, along with meticulous tuning of both computation and memory access. It entails techniques like warp scheduling, fine-grained data layouts, and even PTX-level optimizations. Second, the scarcity of open-source kernels further elevates the barrier to high-performance kernel development, which compels designers to devote extensive time to exploring diverse optimization strategies. Moreover, the rapid iterative modern GPUs and applications inevitably render kernel optimization a challenging and inherently labor-intensive task. Fortunately, recent emerging LLMs have shown promise in automatic kernel generation. Many efforts have attempted to enhance LLM through reflection ( 2 ) , Retrieval-Augmented Generation (RAG) ( 13 ) , or Reinforcement Learning (RL) ( 4 ; 15 ) . However, most existing works remain focused on functional correctness, with significantly lower performance than manually written libraries. This limitation stems from their limited agentic flows, which merely rely on one-shot queries or inefficient reflections. For better exploitation of LLMs, integrating evolutionary algorithms with LLM agents is considered a viable path toward higher-order intelligence ( 18 ; 16 ; 11 ) . For GPU kernel, several works ( 13 ; 6 ) explore crossover prompting techniques to generate new kernels from selected parent kernels, thus evolving kernels’ performance across multiple generations. However, kernels optimized by existing frameworks still remain suboptimal ( 1 ) , exhibiting few basic optimization strategies. We identify that current kernel evolution frameworks suffer from multi-dimensional mismatches between the high-level evolutionary algorithm and low-level kernel code: (1) Mismatched crossover representation: Conventional crossover prompting is operated at the code level, where optimization strategies are implicitly crossed. Such kernel-to-kernel prompting makes LLM to traverse an extended reasoning chain spanning strategy identification, strategy combination, and kernel synthesis. This often yields ineffective optimizations and drops previously attained gains, especially as the kernel’s complexity grows. (2) Mismatched fitness representation: The fitness function, based solely on performance, exhibits weak semantic correlation with the kernel code. This prevents LLM from accurately pinpointing bottlenecks among numerous profiling reports, thus resulting in trivial optimizations. (3) Mismatched population initialization: The initial population is represented by a sparse collection of kernel codes. It provides inadequate coverage of the entire optimization strategy space. Consequently, the optimization is prone to premature convergence to local optima. To address these challenges, this work introduces Strategy as an intermediate representation and proposes a Strategy-Coordinated multi-agent framework named cuPilot. Key contributions are: • A Strategy-Coordinated Evolution (SCE) algorithm is proposed, which decouples the long reasoning chain of conventional crossover prompting into strategy-level crossover and strategy-to-kernel translation. This enables LLM to optimize under explicit strategy guidance, substantially improving the viability and performance of generated kernels. • A roofline-guided prompting method is applied. By first positioning the target kernel on the GPU’s roofline model, the LLM determines if it is compute-bound, memory-bound, or in a middle-zone, thus guiding the prompting for better strategy generation and kernel refinement. • A strategy-level population initialization method is proposed, which establishes an external strategy pool and learn optimization strategies from the historical data via RAG. It enhances the LLM’s strategy generation capability and achieves a higher rate of evolutionary convergence. • Experiment on the 100 kernels of KernelBench ( 19 ) , cuPilot achieves 3.09\times speedup over PyTorch. The case study on GEMM tasks showcases sophisticated optimizations with high utilization of critical hardware units. The generated high-performance kernels are open-sourced. 2. Background 2.1. CUDA Programming To achieve general-purpose GPU programming, NVIDIA developed CUDA (Compute Unified Device Architecture) as a parallel computing platform and programming model ( 9 ) . CUDA extended the C/C++ language to allow developers to leverage the massive parallel computing capabilities of GPUs. CUDA framework unlocks sophisticated control over GPU threads, memory hierarchies, and synchronization. Its programming abstraction also organizes threads into a hierarchical structure of warps, blocks, and grids. This capability renders it indispensable for accelerating computationally intensive tasks involving large-scale matrix and tensor operations ( 12 ) . Furthermore, CUDA’s ecosystem offers highly optimized libraries like cuBLAS and cuDNN ( 7 ) , which significantly enhance application performance for linear algebra and deep neural networks. Therefore, the optimization of CUDA kernels is pivotal for effectively addressing the demands of complex modern applications. 2.2. LLM-Assisted CUDA Kernel Generation Hand-optimization of CUDA kernels is a challenging and time-consuming task that requires designers to possess deep, multifaceted expertise in algorithms and hardware architecture. During the optimization process, designers rely heavily on extensive feedback to guide subsequent manual adjustments. This includes compiler feedback, reports from profiling tools, details of different architectures and instruction sets. These effort-intensive tasks significantly hinder the efficiency of kernel development, a challenge that is particularly acute given the rapid iteration of modern GPUs and applications. An example is FlashAttention kernel, whose adaptation to NVIDIA’s Hopper architecture spanned nearly five years ( 20 ) . Therefore, there is a pressing need for automated kernel development to reduce migration costs. Advances in code-generation LLMs ( 5 ; 17 ; 27 ; 28 ; 23 ; 21 ) have led researchers to leverage them for accelerating GPU kernel development. Some works have sought to improve LLM-generated kernels by integrating reinforcement learning (RL) frameworks ( 4 ; 15 ) , training dedicated models ( 14 ) , and focusing the optimization objective on specific performance metrics (e.g., SwizzlePerf ( 24 ) utilized L2 cache hit rate to guide LLM). Beyond these efforts, a common and viable strategy employs an evolutionary framework based on multi-agent systems. For instance, CUDA-LLM ( 6 ) introduces the FSR framework, which guides LLMs to produce hardware-aware, high-performance CUDA kernels through the joint optimization of compilation and runtime performance. Furthermore, AI CUDA Engineer ( 13 ) incorporates genetic algorithms into the LLM systems. It treats each kernel as an individual candidate and performs crossover prompting across generations of evolution, making a viable path for deep optimization of CUDA kernels. Figure 1. The impact of parent kernels’ complexity on generated child kernels during conventional crossover prompting. nothing 3. Motivation While existing works focus on

Read the complete text