KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu · 2026
Read KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels by Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04956.txt.
Introduction
Han Wang and five co-authors build KernelBenchX, a benchmark for judging GPU kernels written by language models, asking where that skill breaks down and why. Their suite holds one hundred seventy-six tasks across fifteen categories, scored for both correctness and hardware efficiency. Five representative generation methods are compared systematically.
Task structure mattered more than method design, they find, with fusion tasks failing across all methods and maths tasks solved reliably. Stepwise refinement raised correctness but lowered average speed, and nearly half of correct kernels still ran slower than a plain baseline. Quantisation tasks went entirely unsolved, exposing a systematic weakness in numerical code generation.
Catalog record
| CREATOR | Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04956
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu Abstract LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We present KernelBenchX , a benchmark designed to answer this question through category-aware evaluation of correctness and hardware efficiency across 176 tasks in 15 categories. Our systematic comparison of five representative methods yields three main findings. First, task structure determines correctness more than method design. Category explains nearly three times more variance in semantic correctness than method (9.4% vs. 3.3% explained deviance), and 72% of Fusion tasks fail across all five methods while Math tasks are solved consistently. Second, iterative refinement improves correctness, but not performance. Across GEAK iterations, compile rate rises from 52.3% to 68.8% while average speedup declines from 1.58 × 1.58\times to 1.44 × 1.44\times ; newly rescued kernels consistently underperform persistently correct ones ( 1.16 × 1.16\times vs. 1.58 × 1.58\times speedup in round 0 → \to 1). Third, correctness does not imply efficiency. 46.6% of correct kernels are slower than the PyTorch eager baseline, and cross-hardware speedup variance reaches 21.4 × 21.4\times . Besides, quantization remains completely unsolved (0/30 successes) despite non-trivial compilation rates, revealing systematic misunderstanding of numerical computation contracts rather than surface-level syntax errors. These findings suggest that future progress depends on handling global coordination, explicitly modeling numerical precision, and incorporating hardware efficiency into generation. The code is available at https://github.com/BonnieW05/KernelBenchX https://github.com/BonnieW05/KernelBenchX 1 Introduction Background. GPU kernel efficiency has become a central bottleneck in large-scale machine learning workloads 10 ; 27 ; 20 ; 26 . Prior work such as DeepSeek-V3 8 demonstrates that at scale, competitive performance depends not only on model architecture but critically on kernel efficiency. This has motivated a growing body of work using LLMs to automate Triton 11 kernel generation 17 , ranging from training-based methods (AutoTriton 6 , TritonRL 16 ) to agent-based iterative systems (GEAK 12 , STARK 2 ) to search-and-reasoning approaches (KernelEvolve 7 , ReGraph 3 ). Accompanying this methodological progress, several benchmarks have been proposed. KernelBench 9 provides multi-level evaluation from operators to end-to-end models. TritonBench 5 focuses on Triton kernels with a dual-channel testing framework. MultiKernelBench 14 adds cross-platform evaluation, and Robust-KBench 4 emphasizes robustness against misleading performance gains. Limitations. Despite recent progress, two fundamental questions about LLM-based kernel generation remain unresolved. First, the capability boundary is not well described: we do not yet know which types of tasks current methods handle reliably, which consistently fail, and why. Second, the role of iterative refinement is not well understood: it is unclear whether different strategies improve compilation, correctness, or performance, and to what extent. However, existing benchmarks are not designed to answer these questions, due to their unresolved task categories, insufficient correctness verification, and limited evaluation of efficiency. Our Method. To address these limitations, we propose KernelBenchX , built on TritonBench-T and extended in three directions: (1) a robust two-stage correctness protocol that rejects implementations passing output comparison by chance; (2) a unified 15-category taxonomy together with quantization and multi-precision task extensions, enabling fine-grained structural analysis; and (3) hardware-efficiency metrics beyond runtime. Experimental Overview. Using KernelBenchX , we conduct a systematic comparison of representative Triton kernel generation methods. Beyond aggregate benchmark results, we analyze method behavior across task categories, correctness outcomes, and efficiency metrics under a unified evaluation pipeline. We also collect error-correction and optimization pairs from the generation process for future training and inference-time improvement. Contributions. Our main contributions are as follows: • We introduce KernelBenchX , a benchmark for Triton kernel generation with category-aware evaluation of correctness and hardware efficiency. • We conduct a systematic comparison of five representative methods under a unified pipeline. • We identify three empirical findings that characterize the capability boundary of LLM-based kernel generation and provide mechanistic analysis for each. • We release error–correction and optimization pairs collected during evaluation to support future training and inference. 2 Preliminary 2.1 Hardware Efficiency Metrics Hardware efficiency describes how effectively a kernel uses the available hardware resources. Beyond runtime, it provides a more direct view of whether performance is close to the hardware limit. For a kernel k , let T(k) denote the measured runtime, B(k) the total bytes moved, and F(k) the total floating-point operations. We define the achieved bandwidth and achieved throughput as \mathrm{BW}(k)=\frac{B(k)}{T(k)},\qquad\mathrm{TP}(k)=\frac{F(k)}{T(k)}, (1) reported in GB/s and TFLOPS respectively. To enable comparison across hardware, we normalize these quantities by their corresponding peak values: \mathrm{IOU}(k)=\frac{\mathrm{BW}(k)}{\mathrm{BW}_{\max}},\qquad\mathrm{MFU}(k)=\frac{\mathrm{TP}(k)}{\mathrm{TP}_{\max}}. (2) IOU measures memory bandwidth utilization, while MFU measures compute utilization. They are complementary, as some kernels are memory-bound while others are compute-bound. 2.2 Compared Methods We compare five methods spanning key design axes: general-purpose versus specialized models, iterative refinement and domain-specific training. AutoTriton 6 is trained for Triton programming through supervised fine-tuning and reinforcement learning. We evaluate it in single-pass generation with its native prompting. GEAK 12 is an agentic framework with generator, evaluator, reflector, and optimizer modules. We use DeepSeek-V3.2-Chat as the base model, run three iterations at temperature 1.0, generate four candidates per round, and retain five best implementations as context in each round. KernelAgent 13 is a multi-agent system based on a generate–verify–refine workflow, also using DeepSeek-V3.2-Chat. We bypass its Fuser pipeline (not applicable to single-operator tasks) and use its core generation API with 3 parallel workers, up to 5 refinement rounds each, at temperature 0.4. Claude is a strong general-purpose model evaluated in single-pass generation mode. DeepSeek-Coder 8 is a general-purpose code model serving as a zero-specialization baseline. It is evaluated in single-pass generation mode. 3 Methodology 3.1 System Pipeline Figure 1 illustrates the full evaluation pipeline. For each task, KernelBenchX provides a unified description including function interface, reference implementation, and task-specific constraints. Each method generates a candidate kernel through its dedicated adapter. All kernels are then evaluated by the same pipeline, with intermediate logs retained for error analysis. Figure 1: Multiple methods generate Triton kernels from a shared task specification, and are evaluated under a unified pipeline measuring correctness, efficiency, and code quality. 3.2 Benchmark Design 3.2.1 Task Organization KernelBenchX contains 176 tasks across 15 categories: Activation, Convolution, Fusion, Index, LinearAlgebra, Loss, Math, MatrixMultiply, Normalization, Optimizer, Pooling, Quantization, Random, Reduce, and SpatialOps. Categories were assigned based on computational structure rather than operator type, enabling comparison of tasks with similar parallel execution requirements. The benchmark includes two main coverage extensions beyond TritonBench-T. First, selected fp16, bf16, and int8 multi-precision variants test whether methods can generate kernels under low-precision constraints. Second, six quantization tasks in W8A8 and W4A16 settings test whether methods can implement manual quantization logic—including scale computation, explicit casting, and dequantization—without relying on high-level APIs. 3.2.2 Correctness Protocol KernelBenchX uses a two-stage correctness protocol designed to reject implementations that pass simple output comparisons by chance. Call Accuracy checks whether generated code can be imported, compiled, and called correctly, and whether it satisfies task-level constraints. For quantization tasks, a static checker additionally rejects forbidden high-level quantization APIs and verifies the presence of manual quantization logic including scale computation and explicit casting operations. Execution Accuracy checks whether outputs match the reference implementation across multiple input distributions. Each task is evaluated under two