Hunting CUDA Bugs at Scale with cuFuzz Journal: PACMPL Volume: 10 OOPSLA1 123 4 DOI: 10.1145/3798231 CCS: Software and its engineering Software testing and debugging CCS: Security and privacy Software security engineering CCS: Computer systems organization Heterogeneous (hybrid) systems

Mohamed Tarek Ibn Ziad, Christos Kozyrakis · 2026

Read Hunting CUDA Bugs at Scale with cuFuzz Journal: PACMPL Volume: 10 OOPSLA1 123 4 DOI: 10.1145/3798231 CCS: Software and its engineering Software testing and debugging CCS: Security and privacy Software security engineering CCS: Computer systems organization Heterogeneous (hybrid) systems by Mohamed Tarek Ibn Ziad, Christos Kozyrakis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12485.txt.

Introduction

Ibn Ziad and Kozyrakis hunt CUDA bugs at scale with cuFuzz, the first CUDA-oriented fuzzer. GPU programs mix host and device code, breeding memory-safety and concurrency bugs. Prior GPU fuzzing hit false positives, missing device coverage, and clashes between coverage and sanitizers. It fuzzes whole programs, merges NVBit device coverage with host coverage, and splits sanitizers into separate processes. It found 43 unknown bugs in 14 CUDA programs, 19 in commercial libraries. Gains in covered edges and unique inputs beat baselines, above all on closed-source targets.

Catalog record

CREATORMohamed Tarek Ibn Ziad, Christos Kozyrakis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93298
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hunting CUDA Bugs at Scale with cuFuzz Journal: PACMPL Volume: 10 OOPSLA1 123 4 DOI: 10.1145/3798231 CCS: Software and its engineering Software testing and debugging CCS: Security and privacy Software security engineering CCS: Computer systems organization Heterogeneous (hybrid) systems Mohamed Tarek Ibn Ziad, Christos Kozyrakis Abstract . GPUs play an increasingly important role in modern software. However, the heterogeneous host-device execution model and expanding software stacks make GPU programs prone to memory-safety and concurrency bugs that evade static analysis. While fuzz-testing, combined with dynamic error checking tools, offers a plausible solution, it remains underutilized for GPUs. In this work, we identify three main obstacles limiting prior GPU fuzzing efforts: (1) kernel-level fuzzing leading to false positives, (2) lack of device-side coverage-guided feedback, and (3) incompatibility between coverage and sanitization tools. We present cuFuzz, the first CUDA-oriented fuzzer that makes GPU fuzzing practical by addressing these obstacles. cuFuzz uses whole program fuzzing to avoid false positives from independently fuzzing device-side kernels. It leverages NVBit to instrument device-side instructions and merges the resultant coverage with compiler-based host coverage. Finally, cuFuzz decouples sanitization from coverage collection by executing host- and device-side sanitizers in separate processes. cuFuzz uncovers 43 previously unknown bugs (19 in commercial libraries) across 14 CUDA programs, including illegal memory accesses, uninitialized reads, and data races. cuFuzz achieves significantly more discovered edges and unique inputs compared to baseline approaches, especially on closed-source targets. Moreover, we quantify the execution time overheads of the different cuFuzz components and add persistent-mode support to improve the overall fuzzing throughput. Our results demonstrate that cuFuzz is an effective and deployable addition to the GPU testing toolbox. cuFuzz is publicly available at https://github.com/NVlabs/cuFuzz/ . † † cc-license: by 1. Introduction Graphics Processing Units (GPUs) play a critical role in modern computing. They serve as core components across diverse fields, from medical image processing ( 25 ) to large language models ( 39 ) and quantum computing ( 41 ) . As GPUs become larger and more complex, programming them becomes increasingly challenging and error-prone. Recent studies show that memory safety issues, plaguing CPUs for decades ( 42 ) , also affect GPUs ( 32 ; 11 ; 36 ; 37 ) . While there are excellent tools to find memory safety bugs in GPU programs either statically ( 2 ; 3 ) or dynamically ( 29 ; 18 ; 43 ) , these tools have their own limitations. Static tools suffer from high false positive rates whereas dynamic tools have limited scope—they only detect errors triggered by specific program inputs. When error-triggering inputs are not exercised, dynamic tools miss bugs entirely. A natural solution to this problem is fuzz-testing. Fuzz-testing (or simply fuzzing) is a well-established CPU testing technique spanning decades ( 27 ) . It automatically generates random inputs to trigger unexpected program behaviors like crashes and assertions. Fuzzing proves highly effective for uncovering bugs in C/C++ programs ( 8 ) , yet remains largely unexplored for GPUs. Traditional wisdom viewed GPUs as simple accelerators executing well-structured code requiring minimal testing. This perspective no longer holds. Modern GPUs support numerous features, primitives, and operations, making GPU workloads increasingly error-prone. However, naively applying fuzzing to GPUs faces three key challenges: ➀ Fuzzing granularity . GPU programs consist of host-side functions (running on the CPU) and device-side kernels (running on the GPU). Fuzzing can target whole programs or individual kernels. Prior work explored fuzzing individual device kernels by permuting input arguments to expose runtime bugs ( 22 ) or increase coverage ( 34 ) . However, kernel-level fuzzing generates false alarms by creating invalid input combinations that would never occur in complete program execution. For example, a host-side check can invalidate certain device-side input combinations, which kernel-level fuzzing cannot capture. ➁ Lack of device-side coverage . Coverage information is essential for guiding the fuzzer. State-of-the-art CPU fuzzers (like AFL++ ( 6 ) ) use compiler-level instrumentation to collect edge coverage from control flow transitions in the target program. For GPUs, host-side coverage alone is insufficient as it misses critical device-side kernel behaviors. Further, obtaining device-side coverage presents two challenges. First, host and device code use different compilation toolchains (gcc/clang for host, nvcc/ptxas for device). Second, GPUs heavily rely on closed-source libraries ( 31 ) that cannot be recompiled for coverage instrumentation. A unified approach collecting coverage from both host and device sides is essential for effective GPU fuzzing. ➂ Tool incompatibility . Effective fuzzing requires sanitizers for runtime error detection. While Google’s AddressSanitizer (ASan) ( 38 ) dominates CPU testing, no ASan equivalent exists for CUDA due to closed-source GPU libraries and compilers. NVIDIA’s Compute Sanitizer ( 29 ) serves as the alternative, using dynamic binary instrumentation to detect memory safety violations, data races, and uninitialized memory usage. Unfortunately, these sanitizers are neither compatible with each other nor compatible with the coverage collection tools due to shared GPU runtime API dependencies. Hence, coverage collection and sanitization cannot coexist in the same process. This incompatibility severely limits fuzzing effectiveness for GPU code. For example, prior GPU fuzzing work omits sanitizers entirely ( 34 ; 22 ) . Figure 1. cuFuzz high-level overview showing the integration of AFL++ with NVBit for device-side coverage and various sanitizers (e.g., Compute Sanitizer’s memcheck, racecheck, initcheck) for GPU error detection. \pname{} high-level overview. Our work . We propose cuFuzz, the first end-to-end CUDA-oriented fuzzer ( Fig. 1 ). cuFuzz addresses the aforementioned challenges as follows. For challenge ➀ , cuFuzz operates at the whole program level, avoiding false alarms from kernel-level input permutation while preserving inter-kernel and host-device dependencies. For challenge ➁ , cuFuzz leverages NVBit ( 45 ) , a dynamic binary instrumentation tool, to collect device-side coverage at runtime. cuFuzz merges this with host-side coverage to guide comprehensive fuzzing. For challenge ➂ , cuFuzz decouples sanitization from coverage collection by executing sanitizers in separate processes within the fuzzing loop. Implementation . We implement cuFuzz using AFL++ ( 6 ) and NVIDIA’s NVBit ( 45 ) . We evaluate cuFuzz on 14 diverse CUDA programs from HeCBench ( 15 ; 16 ) and CUDALibrarySamples ( 30 ) . Our workloads span open-source programs and closed-source libraries (nvTIFF, nvJPEG, nvJPEG2000, cuDNN, cuBLAS). We perform a thorough evaluation of cuFuzz to assess its bug finding capabilities, edge and input coverage, performance, sanitization, and efficiency. Results . From a bug-finding perspective, at the time of writing, cuFuzz discovered 43 previously unknown bugs, including 19 in production libraries. These bugs range from memory safety violations (host-side heap overflows, device-side illegal memory accesses) to uninitialized memory reads and data races. We reported all bugs to the corresponding developers who acknowledged them. So far, 40 of 43 bugs have been fixed by the libraries’ maintainers in latest releases. To better understand cuFuzz’s individual component contributions, we conduct isolated fuzzing campaigns with only a subset of cuFuzz’s components enabled. We observe that cuFuzz achieves significantly more discovered edges and unique inputs compared to vanilla AFL++, especially on closed-source libraries. Additionally, cuFuzz’s ability to run various incompatible sanitizers allows it to discover more bugs than AFL++. Furthermore, we quantify cuFuzz’s execution overheads by measuring the throughput of its individual components. We observe that device-side coverage collection (NVBit) reduces throughput by 67%

Read the complete text