PBT-Bench: Benchmarking AI Agents on Property-Based Testing
Lucas Jing, Xinqi Wang, Liao Zhang, Simon S. Du · 2026
Read PBT-Bench: Benchmarking AI Agents on Property-Based Testing by Lucas Jing, Xinqi Wang, Liao Zhang, Simon S. Du (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15229.txt.
Introduction
Lucas Jing and three others offer PBT-Bench, a benchmark for property-based testing skill in AI agents. Old benchmarks ask for a bug-copying test or a fix, not the distinct skill of drawing a semantic invariant from documentation and crafting an input strategy that exposes breaks. PBT-Bench holds 100 problems across 40 Python libs with 365 planted bugs, layered from single-constraint boundary bugs to state-based, cross-function protocol violations. Bug recall under property-guided prompting ranges from 42.1 to 83.4 percent across eight models.
Catalog record
| CREATOR | Lucas Jing, Xinqi Wang, Liao Zhang, Simon S. Du |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67263 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PBT-Bench: Benchmarking AI Agents on Property-Based Testing Lucas Jing, Xinqi Wang, Liao Zhang, Simon S. Du Abstract Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing : deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench , a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library’s documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1–L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning. Code is publicly available at https://github.com/ElliotXinqiWang/PBTbench and the dataset at https://huggingface.co/datasets/pbtbench-team/pbt-bench . 1 Introduction Modern language models pass most code generation benchmarks when the expected behavior is stated concretely through input–output pairs. They do less well when the specification is abstract : a documented invariant that must hold across an entire input distribution rather than on any particular example. Property-based testing (PBT), introduced by QuickCheck ( 4 ) and popularized in Python by Hypothesis ( 16 ) , is a testing methodology in which a programmer specifies (i) a property — a predicate that should hold universally — and (ii) a strategy — a typed, composable random-input generator — for sampling from the intended input distribution. The framework then searches for inputs that violate the property and shrinks any counterexample to a minimal failing form. A primer on Hypothesis strategies is in Appendix A.2 . A minimal example of PBT is shown in Figure 1 . We consider a buggy implementation of quicksort ( qsort ) that omits elements equal to the pivot at each recursive step. We verify two properties: the output is sorted, and it is a permutation (i.e., a multiset equivalent) of the input. We use the built-in lists strategy to sample random integer lists as inputs, which can generate cases with duplicate pivot values and thus trigger the assertion failure. PBT provides greater coverage through random input exploration and expresses specifications as machine-checked, living documentation ( 24 ) . However, writing a good property test is harder than a unit test ( 24 ; 8 ) : the author must abstract from concrete outputs to invariants and design an input distribution that targets likely violations—a barrier that LLMs are naturally positioned to lower. Recent benchmarks have begun to probe LLMs’ testing abilities, but none isolates the PBT skill. SWE-bench ( 11 ) rewards patch synthesis given a bug report; SWT-Bench ( 20 ) rewards test synthesis given a bug report; TestExplora ( 18 ) rewards test synthesis from documentation, but over a mined distribution of real-world pull requests that is dominated by bugs whose fixes include dedicated unit tests, I/O coupling, or integration-level failures. Terminal-Bench ( 17 ) evaluates agents end-to-end on terminal tasks with hand-crafted solutions but does not target any specific testing skill. Crucially, even when existing benchmarks withhold explicit bug reports, they only require agents to write finite, concrete test cases. None evaluates the conceptual leap unique to PBT: translating an abstract specification into universal invariants and programmatic input distributions. ⬇ 1 from hypothesis import given , strategies as st 2 3 def qsort ( xs ): 4 if not xs : return [] 5 p = xs [0] # pivot element at each recursion step 6 return qsort ([ x for x in xs [1:] if x < p ]) + [ p ] + \ 7 qsort ([ x for x in xs [1:] if x > p ]) # BUG: ignores elements equal to pivot 8 9 @given ( st . lists ( st . integers ())) 10 # Built-in strategy generating arbitrary integer lists 11 def test ( xs ): 12 ys = qsort ( xs ) 13 # properties: sorted output + multiset equivalence with input 14 assert is_sorted ( ys ) and same_multiset ( xs , ys ) Figure 1 : A Hypothesis property test exposing a quicksort bug: @given(st.lists(st.integers())) generates inputs with duplicates that trigger the missing-pivot-element defect, caught by the sortedness and multiset-equivalence properties. We argue that evaluating PBT as a distinct skill requires curated — not mined — bugs, because not every bug is PBT-testable. A bug is PBT-testable only if (a) a semantic invariant of the API is violated, (b) the violation manifests under some expressible Hypothesis strategy without environmental mocks, and (c) the violation is deterministic under random input sampling. In practice, many real-world pull request bugs do not satisfy at least one of these conditions — common examples include UI rendering failures, database state dependencies, network ordering effects, and type errors that lack a corresponding documented invariant. A benchmark mined from PR history therefore conflates PBT skill with the agent’s ability to navigate around non-PBT-testable cases. Contributions. 1. PBT-Bench, a suite of 100 property-based testing problems across 40 real Python libraries (serialization, data structures, date–time, numerics, type systems, state machines, parsing), each designed and verified to satisfy the PBT-testability conditions above. Every bug is accompanied by a ground-truth Hypothesis strategy, a precise trigger condition, and a three-level difficulty label (L1–L3). To the best of our knowledge, this is the first benchmark that evaluates coding agents’ ability to use PBT for bug detection in program libraries. 2. An automated, containerized F \to P (fail on buggy, pass on fixed) harness that evaluates each test function independently per injected bug, so that a test file covering multiple bugs is scored per-bug and not globally. 3. A large-scale evaluation: eight contemporary LLMs (Claude Sonnet 4.6, DeepSeek V3.2, Gemini 3 Flash, GLM 5.1, Grok 4.1 Fast, Qwen 3.6 Plus, Qwen 3.5-30B-A3B, Step 3.5 Flash), each run in two prompting regimes (open-ended baseline and explicit Hypothesis scaffolding) for three independent runs per cell, yielding 4{,}800 agent trajectories. 4. Three findings that appear specific to PBT evaluation: (i) the baseline-vs-PBT prompt gap is largest for models with weaker baselines (+24.5pp, +22.9pp, +20.3pp) and smaller or inverted for models with stronger baselines (+6.7pp for Sonnet 4.6; -3.2 pp for DeepSeek V3.2; -8.0 pp for Grok 4.1 Fast), consistent with scaffolding substituting for a capability weaker models lack, rather than complementing a skill stronger models