Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs
Dipankar Sarkar · 2026
Read Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs by Dipankar Sarkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27396.txt.
Introduction
Sarkar measures which test-input choices actually catch bugs in tensor programs, rather than trusting folklore. GPU kernel projects typically test one representative shape and data type, then ship. Seven generation strategies are compared on a corpus of twenty-six operations: sixteen correct controls and ten buggy variants seeded with documented error patterns. Shape sets, data-type mixes, and value distributions vary across strategies. A schema-aware seeded fuzzer on a consumer graphics card runs the experiments, in a software testing study published in 2026.
Boundary-only shape sampling wins on safety: it catches most seeded bugs while raising zero false alarms on the correct controls. Aggressive value sampling catches nearly everything but floods the controls with false positives, because injected not-a-number values trip validators on healthy kernels too. For two bugs in softmax tail masks, the regular strategy catches nothing, while boundary shapes catch most or all.
Catalog record
| CREATOR | Dipankar Sarkar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs Dipankar Sarkar Abstract Test-input generation for tensor kernels is folkloric. Most projects pick a representative shape and dtype, run a fixed-shape allclose-style check, and ship. We make the choices explicit and measure them. Using the gpuemu op-schema-aware seeded fuzzer [ 8 ] , we evaluate seven test-generation strategies across a 26-op corpus (16 correct controls and 10 LLM-style buggy variants seeded with documented transcription patterns) on an RTX 3060 GPU instance. Strategies vary the shape candidate set, the dtype mix, and the input value distribution. We report each strategy on two axes: bug recall and control false-positive (FP) rate. Boundary-only shape sampling is the operationally safe winner: 78% recall on the 10 buggy kernels with 0% FP on the 16 controls. Adversarial value sampling reaches higher recall (99%) but inflates control FP to 94% because the strategy injects NaN and Inf inputs and the validator’s NaN check fires on every kernel that propagates them, not only on buggy kernels. On the two softmax tail-mask bugs the “regular” strategy (no boundary shapes) catches 0%, while boundary raises recall to 100% and 62% respectively. That gap is the clearest single signal in the data. The corpus result is about which seeded bug patterns each strategy catches, not about the bug rate of any specific deployed LLM. 1 Introduction Generated GPU kernels from recent benchmarks [ 4 , 10 , 6 , 2 ] are tested by an oracle that takes inputs from somewhere. Where is rarely documented and almost never compared. The “regular shape” default (one shape per operator, one dtype, uniform random values) is the convenient choice, but it has two empirically measured failure modes. Shape-dependent bugs. Tail-mask leak in reductions and accumulator overwrite in matmul only surface at specific boundary shapes (e.g., H=3 , K=1 ). A test at H=256,K=16 cannot see them. Magnitude-sensitive bugs. Overflow with extreme values, and special-case handling for zero, Inf, and NaN, only surface under input distributions that include those values. This paper measures both gaps and quantifies what each strategy buys. The contributions are four. 1. A strategy taxonomy for tensor-kernel test-input generation, exposed as switchable knobs on the gpuemu fuzzer [ 8 ] . The knobs are the shape candidate set (boundary, regular, or default mix), the dtype set, and the value distribution (Uniform, NaNInjected, or Adversarial). 2. A controlled ablation across 26 ops, 7 strategies, and 8 iterations per (strategy, kernel), 1,456 cases on a single RTX 3060. 3. A ranked table that reports each strategy on both recall (on the 10 buggy kernels) and control FP rate (on the 16 correct kernels). Adversarial reaches 99% recall but 94% FP. Boundary reaches 78% recall at 0% FP. The “regular” strategy loses an entire bug class (0% recall on tail-mask bugs). 4. A measured explanation of the recall-FP trade-off for the adversarial and NaN-injected strategies. Both inject non-finite inputs, both inflate FP, and the inflation traces to the validator’s NaN check rather than to genuine output divergence. 2 Related Work Boundary-value testing. A classical software-testing principle dating to Myers [ 3 ] . The highest defect density sits at the boundaries of input partitions. Modern coverage-guided fuzzers (AFL, libFuzzer) implement variants of this. For tensor kernels the analogue is shape boundaries (1, prime, power of two \pm 1 ) and value boundaries (0, subnormal, near fp-max, \pm Inf, NaN). DL library fuzzing. FreeFuzz [ 11 ] , DocTer [ 12 ] , DeepREL [ 1 ] , and NablaFuzz [ 13 ] all fuzz the API layer of TensorFlow, PyTorch, and JAX. Their generation strategies are mostly value mutational and they evaluate at the API level, not the kernel level. Coverage-guided variants [ 5 ] focus on syntactic coverage of the API surface, not the operator’s input domain. None of them publish a per-strategy ablation of the kind below. Kernel-level metamorphic testing. A few works derive metamorphic relations for specific operators (e.g., softmax shift-invariance), but the practice has not propagated to the LLM-kernel ecosystem. LLM-kernel benchmarks. The benchmarks [ 4 , 10 , 6 , 2 , 9 ] are usually one-shape, one-dtype. KernelBench’s [ 4 ] correctness oracle is torch.allclose on the reference shape. Without varied inputs an entire class of bugs is invisible. This is the gap we measure. 3 Method 3.1 Strategy parameters A strategy is a triple ( op_schema , dtypes , value_distribution ) applied uniformly to every (op, iter) under the strategy. op_schema a per-input shape generator. Each op ships a native schema (mixed boundary and regular candidates). A strategy can override the candidate set for any dim. For example boundary restricts H to \{1,3,7\} and regular restricts H to \{128,256,512\} . dtypes a list of dtypes to round-robin across iters. value_distribution how the fuzzer fills tensor values. Uniform samples U[-10,10] (default). NaNInjected replaces 5% of float elements with NaN, \pm Inf, or 0. Adversarial samples equal-weight across five dtype-aware buckets: 0 ; very small (near the dtype’s tiny value, subnormal for fp32 and fp16); large (max divided by 10 ); wide-uniform U[-10^{3},10^{3}] ; and non-finite ( \pm Inf, NaN). 3.2 The seven strategies evaluated Table 1 lists the seven strategies. All seven share the same operator corpus and the same per-op tolerances. Only the test-input generator changes. Table 1: The seven strategies evaluated in the ablation. strategy op_schema dtypes value_distribution default native native Uniform boundary H\in\{1,3,7\} native Uniform regular H\in\{128,256,512\} native Uniform single_dtype_f32 native {float32} Uniform single_dtype_f16 native {float16} Uniform nan_injected native native NaNInjected adversarial native native Adversarial 3.3 Bookkeeping The driver ( drivers/p3_strategies.py ) records per (strategy, op, iter) the verdict, dtype, shape, layout, failure_kind , and error_stats . It also records time-to-first-failure (in seconds) per (strategy, op) as a wall-clock efficiency metric. 3.4 Assumptions The ablation depends on four assumptions. 1. The 16 controls are correct kernels (human-written Triton or numpy stand-ins). The companion paper [ 8 ] establishes their correctness on the gpuemu oracle. 2. The 10 buggy variants are author-seeded with documented LLM transcription patterns. They are not pulled from real LLM-generated outputs. 3. Tolerances are fixed per (op, dtype) across all seven strategies. The companion paper [ 7 ] addresses the tolerance question separately. Mixing the calibration step in here would conflate two effects. 4. The Python client decodes received tensors as contiguous, so layout-only strategies (transposed, strided) are nominal at the client boundary. The daemon-side fuzzer correctly varies strides but the kernel sees contiguous data. 4 Evaluation Setup. RTX 3060, image pytorch/pytorch:2.4.0-cuda12.4-cudnn9-devel . 7 strategies \times 26 ops \times 8 iters =1{,}456 cases. Run id run-20260611-101922-4dcac1 on Backblaze B2. Headline: recall and control FP per strategy. Each strategy ran the same 10 buggy kernels and the same 16 controls at 8 iterations per (strategy, kernel). That gives 80 buggy cases and 128 control cases per strategy. Table 2 reports both axes. Table 2: Per-strategy recall (10 buggy \times 8 iters =80 ) and control FP rate (16 correct \times 8 = 128). strategy bug recall control FP adversarial 79/80 (99%) 120/128 (94%) nan_injected 75/80 (94%) 93/128 (73%) boundary 62/80 (78%) 0/128 (0%) single_dtype_f32 59/80 (74%) 0/128 (0%) single_dtype_f16 59/80 (74%) 3/128 (2%) default 57/80 (71%) 0/128 (0%) regular 51/80 (64%) 0/128