Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code Empirical Analysis of 3,500 Code Artifacts Across Seven Large Language Models Using Z3 SMT Solver, Ablation Studies, and Static Tool Comparison

Dominik Blain, Founder, Maxime Noiseux, Co-Founder · 2026

Read Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code Empirical Analysis of 3,500 Code Artifacts Across Seven Large Language Models Using Z3 SMT Solver, Ablation Studies, and Static Tool Comparison by Dominik Blain, Founder, Maxime Noiseux, Co-Founder (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05292.txt.

Introduction

Blain and Noiseux present Broken by Default, a formal study of 3,500 code artifacts from seven language models across 500 security prompts. Each artifact is checked with the Z3 solver via COBALT, producing mathematical witnesses. Across models, 56 percent of artifacts contain at least one COBALT vulnerability; of these, 1,055 are formally proven via Z3 witnesses. GPT-4o leads at 62 percent, grade F, while Gemini 2.5 Flash is best at 48 percent, grade D, and no model earns better than D. Six industry tools combined miss 98 percent of Z3-proven findings.

Catalog record

CREATORDominik Blain, Founder, Maxime Noiseux, Co-Founder
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27621
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05292

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code Empirical Analysis of 3,500 Code Artifacts Across Seven Large Language Models Using Z3 SMT Solver, Ablation Studies, and Static Tool Comparison Dominik Blain, Founder, Maxime Noiseux, Co-Founder Abstract AI coding assistants are now used to generate production code in security-sensitive domains, yet the exploitability of their outputs remains unquantified. We address this gap with Broken by Default : a formal verification study of 3,500 code artifacts generated by seven widely-deployed LLMs across 500 security-critical prompts (five CWE categories, 100 prompts each). Each artifact is subjected to the Z3 SMT solver via the COBALT analysis pipeline, producing mathematical satisfiability witnesses rather than pattern-based heuristics. Across all models, 55.8% of artifacts contain at least one COBALT-identified vulnerability; of these, 1,055 are formally proven via Z3 satisfiability witnesses (Z3 SAT). GPT-4o leads at 62.4% (grade F); Gemini 2.5 Flash performs best at 48.4% (grade D). No model achieves a grade better than D. Six of seven representative findings are confirmed with runtime crashes under GCC AddressSanitizer. Three auxiliary experiments on a 50-prompt sub-corpus show: (1) explicit security instructions reduce the mean rate by only 4 points to 60.8%, leaving four of five models at grade F; (2) six industry tools combined flag 7.6% of artifacts and miss 97.8% of Z3-proven findings—a structural gap, not a configuration issue; and (3) models identify their own vulnerable outputs 78.7% of the time in review mode yet generate them at 55.8% by default, revealing a persistent generation–review asymmetry. Formal SMT verification is the only methodology that can establish ground-truth exploitability of the vulnerability patterns that dominate AI-generated code. 1 Introduction The rapid adoption of AI-assisted coding tools has fundamentally changed software development workflows. GitHub Copilot reported over 1.8 million paid subscribers by 2024 [ 1 ] . ChatGPT, Claude, and Gemini are routinely used to generate production code across security-sensitive domains including web backends, embedded systems, and cryptographic libraries. Yet the security properties of AI-generated code remain understudied. Prior work [ 2 , 3 , 4 ] has identified categories of vulnerabilities in LLM outputs, but most rely on static pattern matching or manual inspection—techniques that cannot definitively establish exploitability. Our contribution is methodological: we apply formal verification via the Z3 Satisfiability Modulo Theories (SMT) solver to establish ground-truth exploitability of AI-generated code vulnerabilities. Where a Z3 witness exists, the vulnerability is not a potential issue—it is a proven one, with a concrete input that triggers the fault condition. We further validate selected findings with compiler-instrumented runtime crashes using GCC AddressSanitizer. Our findings (500-prompt v3 corpus, seven models) reveal a mean vulnerability rate of 55.8% across all models. GPT-4o leads at 62.4% (grade F); Gemini 2.5 Flash performs best at 48.4% (grade D)—no model achieves grade C or better. A total of 1,055 findings were formally proven exploitable via Z3 satisfiability witnesses, and 6 of 7 selected vulnerabilities were confirmed with real memory corruption crashes using GCC AddressSanitizer (ASAN). We further conduct three additional experiments on the v1 50-prompt corpus: (1) a secure prompt ablation showing that explicit security instructions reduce the mean vulnerability rate by only 4 percentage points (to 60.8%), leaving four of five models at grade F; (2) a static tool comparison showing that six industry tools combined flag only 7.6% of artifacts, missing 97.8% of formally Z3-proven vulnerabilities; and (3) a generation–review asymmetry experiment showing that models identify their own Z3-proven vulnerabilities 78.7% of the time in review mode (70/89)—yet generate them at 55.8% by default. Our results suggest that current LLMs produce security-deficient code by default, that security instructions are insufficient mitigations, and that formal verification is the only approach capable of establishing ground-truth exploitability at scale. 1.1 Research Questions • RQ1: At what rate do leading LLMs produce vulnerable code when prompted for security-sensitive tasks? • RQ2: How do vulnerability rates and severity distributions differ across models? • RQ3: Can Z3 SMT witnesses be operationalized into real runtime exploits? • RQ4: Do explicit security instructions in the system prompt reduce vulnerability rates? • RQ5: How does COBALT’s detection rate compare to industry-standard static analysis tools? • RQ6: Do models possess the knowledge to detect their own generated vulnerabilities? 1.2 Scope and Limitations This study targets a bounded set of 500 prompt templates (100 per CWE category), each designed to elicit code in a security-sensitive domain. Results reflect model behavior on this prompt corpus as of April 2026 and should not be extrapolated to all possible LLM usage patterns. Models were queried via their production APIs at temperature 0 (deterministic mode). Prompt templates are released alongside this paper. 2 Related Work Pearce et al. (2022) [ 2 ] evaluated GitHub Copilot on 89 scenarios across 18 CWEs and found 40% of suggestions contained vulnerabilities. Their analysis relied on CWE pattern matching without formal exploitability proofs. Sandoval et al. (2023) [ 3 ] studied security of Copilot suggestions for C code, finding that users who accepted suggestions without review introduced more vulnerabilities than those who wrote code from scratch. Tony et al. (2023) [ 4 ] introduced LLMSecEval, a benchmark of 150 natural-language prompts and evaluated 5 LLMs, finding high rates of insecure code generation particularly for injection and buffer-related vulnerabilities. Perry et al. (2023) [ 5 ] conducted a user study where participants using AI assistants wrote significantly more security vulnerabilities than those who did not, despite expressing higher confidence in their code. Meta CyberSecEval (2024) [ 11 ] benchmarks LLMs on generating exploit code and assisting cyberattacks. Their focus is adversarial capability—can the model help an attacker? Our focus is orthogonal: does the model introduce vulnerabilities in everyday code generation, without any adversarial intent? SecurityEval (2022) [ 12 ] provides 130 Python prompts derived from CWE patterns and evaluates GitHub Copilot. Like Pearce et al., evaluation relies on pattern-based analysis. No formal exploitability proof is provided. SVEN (2023) [ 13 ] trains a security vector to steer code generation toward secure outputs. Their setup assumes the vulnerability exists; ours measures baseline generation without steering intervention. Our work differs in four key dimensions: (1) we apply formal Z3 SMT verification rather than pattern matching—producing mathematical proof of exploitability with concrete witness values; (2) we evaluate seven widely-deployed production models simultaneously across identical prompts; (3) we provide runtime exploitability confirmation via ASAN; and (4) we quantify the detection gap against six industry tools, including heavyweight C analyzers and CodeQL, demonstrating that 97.8% of formally proven vulnerabilities are structurally invisible to existing tooling. 3 Methodology 3.1 Benchmark Design We constructed a benchmark of 500 natural-language prompts distributed across five CWE categories (100 prompts each): • MEM (CWE-131/190): Buffer allocation, array indexing, dynamic memory management in C/C++ • INT (CWE-190/195): Integer arithmetic, type casting, signed/unsigned arithmetic • AUTH (CWE-916): Password hashing, authentication token generation, session management • CRYPTO (CWE-327/330): Key generation, random number usage, cipher selection • INP (CWE-89/22/78): SQL construction, file path handling, shell command composition Prompts were designed to be representative of real developer tasks—not adversarial jailbreaks. Example prompt (MEM-01): “Write a C function that reads n integers from a file descriptor and returns them as a dynamically allocated array.” 3.2 Model Selection and API Configuration Seven models were evaluated,

Read the complete text