The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review
Christo Zietsman · 2026
Read The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review by Christo Zietsman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25773.txt.
Introduction
Zietsman argues that AI code review fails when no executable specification exists. Without an outside reference, the writing agent and the reviewing agent reason from the same code. They share training data and tend to fail in the same places. Zietsman sets out three hypotheses. Correlated errors in similar pipelines echo rather than cancel. Executable specs move a task from complex to merely complicated. The flaws specs cannot reach form a bounded residue fit for AI review. His order is specs first, checks second, AI review last.
Catalog record
| CREATOR | Christo Zietsman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38438 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25773
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review Christo Zietsman Abstract The dominant industry response to AI-generated code quality problems is to deploy AI reviewers. This paper argues that this response is structurally circular when executable specifications are absent: without an external reference, both the generating agent and the reviewing agent reason from the same artefact, share the same training distribution, and exhibit correlated failures. The review checks code against itself, not against intent. Three hypotheses are developed. First, that correlated errors in homogeneous LLM pipelines echo rather than cancel, a claim supported by convergent empirical evidence from multiple 2025-2026 studies and by three small contrived experiments reported here. The first two experiments are same-family (Claude reviewing Claude-generated code); the third extends to a cross-family panel of four models from three families. All use a planted bug corpus rather than a natural defect sample; they are directional evidence, not a controlled demonstration. Second, that executable specifications perform a domain transition in the Cynefin sense, converting enabling constraints into governing constraints and moving the problem from the complex domain to the complicated domain, a transition that AI makes economically viable at scale. Third, that the defect classes lying outside the reach of executable specifications form a well-defined residual, which is the legitimate and bounded target for AI review. The combined argument implies an architecture: specifications first, deterministic verification pipeline second, AI review only for the structural and architectural residual. This is not a claim that AI review is valueless. It is a claim about what it is actually for, and about what happens when it is deployed without the foundation that makes it non-circular. This paper develops three interconnected hypotheses about the role of executable specifications in AI-assisted software development. Citations are provided for verification; readers are encouraged to consult the original sources rather than relying on the summaries here. 1 Introduction The AI code review market is growing rapidly, with tools like CodeRabbit, Cursor Bugbot, and GitHub Copilot deployed across tens of thousands of engineering teams. The value proposition is straightforward: AI generates code faster than humans can review it, so use AI to review it. The premise is reasonable. The architecture that follows from it has a structural problem the industry conversation has mostly not examined. The DORA 2026 report, based on 1,110 open-ended survey responses from Google engineers, found that higher AI adoption correlates with higher throughput and higher instability simultaneously. Time saved generating code is re-spent auditing it. The bottleneck has moved from writing code to knowing what to write and verifying that what was written is correct. Deploying more AI at the review stage does not address the structural problem; it adds another probabilistic layer to a pipeline that already lacks a deterministic quality gate. This paper examines that problem in three stages. Section 2 develops the correlated error hypothesis, drawing on recent empirical work on LLM ensemble pipelines. Section 3 develops the Cynefin domain transition hypothesis, grounding the specification-first argument in complexity science. Section 4 proposes a taxonomy of defect classes that executable specifications cannot catch, grounding the residual in oracle problem theory. Section 5 discusses the architecture implied by the combined argument and identifies what remains open. 2 The Correlated Error Hypothesis 2.1 The Structural Problem When an AI coding agent generates code and a separate AI reviewer examines it without an external specification, both agents reason from the same artefact: the code. The reviewer has no ground truth to compare against. It checks the code against itself, not against intent. This has a precise formulation in machine learning theory. In classical ensemble learning, stacking multiple estimators improves reliability under one condition: the estimators must fail independently. If two classifiers share the same training distribution and the same blind spots, combining them does not reduce error. It consolidates it. The joint miss rate of two correlated estimators approaches the miss rate of either one alone, not the product of both (Dietterich 2000; Hansen and Salamon 1990). A code-generating model and a code-reviewing model from the same model family share architecture, training corpus, and reward signal. That is the same class of correlation the independence condition prohibits. They are not two independent estimators. They are two samples from the same prior. 2.2 The Empirical Evidence A 2025 paper studying LLM ensembles for code generation and repair (Vallecillos-Ruiz, Hort, and Moonen, arXiv:2510.21513) coined the term "popularity trap" to describe what happens when multiple models vote on candidate solutions. Models trained on similar distributions converge on the same syntactically plausible but semantically wrong answers. Consensus selection (the default review heuristic in most multi-agent pipelines) filters out the minority correct solutions and amplifies the shared error. Diversity-based selection recovered up to 95% of the gain that a perfectly independent ensemble would achieve. A second paper (Mi et al., arXiv:2412.11014) examined a researcher-then-reviser pipeline for Verilog code generation and found that erroneous outputs from the first agent were accepted downstream because the agents shared the same training distribution and lacked adversarial diversity. Single-agent self-review degenerated into repeating the original errors. A February 2026 paper (Pappu et al., arXiv:2602.01011) showed that even heterogeneous multi-agent teams consistently failed to match their best individual member, incurring performance losses of up to 37.6%, even when explicitly told which member was the expert. The failure mechanism is consensus-seeking over expertise. Homogeneous copies of the same model family make the failure mode worse. A 2025 paper on test case generation (arXiv:2507.06920) found that LLM-generated verifiers exhibited tightly clustered error patterns, indicating shared systematic biases, while human errors were widely distributed. LLM-based approaches produced test suites that mirrored the generating model’s error patterns, creating what the authors call a homogenisation trap where tests focus on LLM-like failures while neglecting diverse human programming errors. This is the correlated error argument confirmed from the test generation side rather than the review side. Jin and Chen (arXiv:2508.12358, 2025) found a systematic failure of LLMs in evaluating whether code aligns with natural language requirements, with more complex prompting leading to higher misjudgement rates rather than lower ones. A March 2026 follow-up (arXiv:2603.00539) confirmed that LLMs frequently misclassify correct code as non-compliant when reasoning steps are added. These papers test reviewers given specifications in prose form; the overcorrection bias they document is a separate failure mode from the correlated miss problem, but both point to the same structural weakness: without a deterministic ground truth, the review is unreliable in both directions. Wang et al. (arXiv:2512.17540, 2025) proposed a specification-grounded code review framework deployed in a live industrial environment and found that grounding review in human-authored specifications improved developer adoption of review suggestions by 90.9% relative to a baseline LLM reviewer. Note: the 90.9% figure refers to adoption rate, not defect detection rate. None of these papers tested the exact configuration the hypothesis describes: a pure generator-then-reviewer pipeline without any external grounding, measuring shared failure modes directly. The contrived experiments in Section 2.5 provide directional evidence for that configuration, subject to the limitations stated there. 2.3 A Constructed Illustration The following example uses a deliberately simple case. Modern frontier models catch classic boundary conditions reliably, as the experiments in Section 2.5 confirm. The purpose here is not to demonstrate