Before the Model Learns the Bug: Fuzzing RLVR Verifiers

Jaideep Ray · 2026

Read Before the Model Learns the Bug: Fuzzing RLVR Verifiers by Jaideep Ray (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01066.txt.

Introduction

Ray fuzzes the graders that train AI with checkable rewards in this 2026 paper. In RLVR, executable checkers replace human taste: math answer tests, JSON validators, code unit tests. The reward is thus partly a software artifact. If the checker is wrong, training can learn the bug itself. The failure grows quietly, since scores still rise.

His framework is light by design. It crafts adversarial completions meant to slip past a weak checker. Each output faces both the suspect verifier and a stricter reference one. Paired verdicts are logged for every case. The report then counts false positives, false negatives, and splits between the two graders. Exploit and uncertainty rates round out the scorecard. The point is timing: catch verifier bugs before the model learns to love them. The paper serves teams who build reward code for reasoning training. A buggy grader, left alone, becomes the lesson the model studies hardest.

Catalog record

CREATORJaideep Ray
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01066

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Before the Model Learns the Bug: Fuzzing RLVR Verifiers Jaideep Ray Abstract Reinforcement learning with verifiable rewards (RLVR) replaces human preference labels with executable reward functions such as math answer checkers, JSON tool-call validators, and code unit-test harnesses. That makes the reward partly a software artifact: if the verifier is wrong, optimization can learn the bug. We study this failure mode with a lightweight verifier-fuzzing framework that generates adversarial completions, compares buggy and stricter reference verifiers, logs paired decisions, and reports false-positive, false-negative, disagreement, exploit, and uncertainty metrics. Across seeded runs, intentionally buggy math, JSON tool-call, and toy code verifiers repeatedly accept incorrect completions, while stricter variants remove the measured false positives on the same generated cases. An open-source replay grounds the diagnostic in existing validators: relative to our stricter final-answer contract, math-verify accepts partial_answer cases, while a SymPy-backed final-answer replay removes those false positives by accepting a narrower language. The jsonschema replay accepts none of the adversarial JSON cases, and its zero coverage identifies format-level non-engagement rather than semantic validation. Hardening ablations, template search, tabular policy gradients, and budgeted black-box querying show that false-positive regions are structured enough to sustain high reward while strict proxy correctness remains low. A HumanEval-style visible-vs-hidden code evaluation reproduces the visible-test failure mode outside the main synthetic fuzzers. These results make verifier reliability a pre-training systems property that can be measured and audited before optimization begins. Keywords: verifiable rewards; RLVR; verifier fuzzing; reward hacking; differential testing; tool-call validation; code evaluation 1 Introduction RLVR is attractive because it turns machine-checkable tasks into scalable post-training signals. A model can be rewarded for producing a final mathematical answer, emitting a schema-valid tool call, or writing code that passes tests, reducing dependence on human labels and making pipelines more reproducible. The systems risk is that the reward is executable software. The verifier parses outputs, extracts answers, validates schemas, compares values, executes tests, enforces timeouts, and decides whether to award reward. Bugs in this stack do not merely add noise: under optimization pressure, they become rewardable failure modes. A verifier that accepts the first number in a response, ignores duplicate JSON keys, or treats printed text as program correctness creates high-reward false positives. This paper asks a practical systems question: can realistic verifier bugs create high-reward false positives that simple search or RL-style optimization can reliably find? In controlled local experiments, yes. We build a verifier-fuzzing prototype for math final-answer verification, JSON tool-call verification, and toy code unit-test verification. We pair intentionally buggy implementations with stricter reference variants, deterministic and held-out fuzzers, JSONL logs, hardening ablations, optimization proxies, and targeted replay against open-source math and JSON validators. The paper focuses on a systems diagnostic rather than a new RL algorithm: verifier reliability is measurable before expensive training begins, so RLVR builders should stress-test reward verifiers before using them as training objectives. This paper makes four contributions: • A practical taxonomy of short, realistic verifier bugs for math, tool-call, and code-reward pipelines. • A reproducible paired verifier-fuzzing workflow that logs buggy-versus-strict outcomes and computes metrics only from saved JSONL records. • Multi-seed evidence across held-out, adaptive, and optimization settings that false-positive regions are structured and exploitable. • An engineering view of verifier hardening, including ablations that localize high-value fixes, an open-source verifier replay that tests existing validators, and an external HumanEval-style code evaluation that validates the visible-test failure mode outside the main synthetic fuzzer. 2 Background and Related Work Reward hacking and specification gaming are long-standing concerns in reinforcement learning. Amodei et al. identify reward hacking as a concrete AI safety problem: an agent may exploit loopholes in an objective rather than satisfy the intended task [ 1 ] . RLVR makes one class of loopholes especially concrete: verifier implementation defects. Recent RLVR work has observed verifier gaming after model training. Helff et al. study LLMs gaming verifiers in inductive reasoning tasks and use isomorphic perturbation testing to expose shortcut behavior [ 2 ] . Khalifa et al. introduce Countdown-Code, a controlled environment where proxy reward can be separated from true task reward [ 3 ] . Our method is complementary. Instead of first training a model and then diagnosing reward hacking, we fuzz the verifier directly as a pre-training diagnostic. Our code setting is related to code-generation evaluation. Prior work shows that insufficient tests can overestimate functional correctness of generated code [ 4 ] . We study the same issue from the reward-function perspective: if an RLVR verifier runs only visible tests or accepts stdout as evidence of correctness, it can reward code that does not implement the intended function. Our JSON tool-call setting is related to LLM tool-use and API-invocation benchmarks, including ToolLLM’s tool-use data and evaluation resources [ 5 ] . Tool-call evaluation often relies on schema-based validation, making it natural to study bugs involving wrong tool names, ignored argument values, duplicate keys, embedded JSON, and unexpected fields. 3 Failure Model and Metrics A verifier receives a task, gold specification, and completion, then returns a reward. The dangerous event is a silent false positive: verifier accepts completion, but completion is semantically incorrect. For a completion c , let A(c) be verifier acceptance and T(c) be true semantic correctness. A silent false positive is A(c)=1\quad\text{and}\quad T(c)=0. False negatives are also undesirable because they reject valid completions, but false positives are more directly connected to reward hacking. They define regions of completion space where optimization receives high reward without solving the task. The policy or search process is not assumed to have source access to the verifier. It only needs repeated reward feedback. This matches ordinary local RLVR training: completions are sampled, evaluated, rewarded, and then sampled more or less often based on the reward. We report the following metrics from saved logs: • False-positive rate: accepted and truly incorrect completions divided by truly incorrect completions. • False-negative rate: rejected and truly correct completions divided by truly correct completions. • Differential disagreement rate: paired buggy and strict verifier decisions differ. • Exploit-candidate rate: buggy verifier accepts and strict verifier rejects. • Reward-correctness gap: mean verifier reward minus strict proxy correctness in optimization experiments. • Exploit rate: sampled completions that are rewarded but truly incorrect according to the workload’s expected-correct label. In optimization tables where only strict-reference decisions are available, we call the analogous measure proxy exploit rate . Throughout the paper, T(c) is the intended semantic label supplied by the workload design. The strict verifier is an operational proxy for T(c) , not a formal proof of correctness. Accordingly, a strict false-positive rate of zero means zero on the generated cases in this controlled evaluation, not zero for all possible completions or deployments. When a denominator is zero, such as a negative-only replay set with no truly correct candidates, we report the corresponding rate as N/A rather than 0.000. 4 Verifier-Fuzzing System The prototype has five stages: 1. Define seed tasks with prompts, gold answers, and verifier-specific gold specifications. 2. Generate adversarial completions by mutation category. 3. Evaluate each completion with paired buggy and strict verifiers. 4. Save one JSONL record per verifier-case result. 5. Analyze only saved logs to produce tables, confidence intervals, and numerator/denominator summaries. Each task is represented as a VerificationExample record. Key fields include the

Read the complete text