We built our security programs on an assumption so foundational that most of us never articulated it: the person writing the code understands what they’re writing. They know the language’s edge cases, the context of the system, the difference between code that compiles and code that’s safe. Our entire review apparatus — peer review, static analysis, penetration testing — was designed around the premise that a knowledgeable human sits at the origin of every code artifact, and our tools exist to catch the mistakes that even knowledgeable humans make.

That assumption is now invalid for a growing share of production code. And the evidence is no longer anecdotal.

A formal verification study published April 5, 2026 — Broken by Default by Dominik Blain and Maxime Noiseux of Cobalt AI — has produced the most methodologically rigorous quantification I’ve seen of AI-generated code insecurity. Not heuristic warnings. Not pattern-matching guesses. Mathematical proof of exploitability, using the Z3 SMT solver to generate concrete exploit inputs for each vulnerability identified.

The top-line number: 55.8% of AI-generated code artifacts contain at least one verified vulnerability. Across seven widely-deployed LLMs. Across 3,500 code samples. With 1,055 vulnerabilities formally proven exploitable via satisfiability witnesses — meaning the solver produced the exact input value that triggers each fault.

This post is my attempt to unpack what that means — not just as a research finding, but as an operational reality for anyone responsible for securing an enterprise software development pipeline in 2026.

From Pattern Matching to Mathematical Proof

The AI code security conversation has been building for years. Pearce et al. (2022) evaluated GitHub Copilot on 89 scenarios and found 40% of suggestions contained vulnerabilities. Perry et al. (2023) ran a controlled user study showing developers using AI assistants wrote significantly more security bugs — while reporting higher confidence in their code. Veracode’s 2025 GenAI Code Security Report tested over 100 LLMs across 80 coding tasks and found a 45% security failure rate, unchanged in their 2026 update.

These studies established the problem. Broken by Default changes the conversation because it establishes the ground truth of exploitability.

Prior work relied on CWE pattern matching — static rules that identify code structures known to be associated with vulnerability classes. This tells you something looks dangerous. It does not tell you whether an attacker can actually trigger the fault, or what input would do it.

The study evaluated seven models — GPT-4o, GPT-4.1, Claude Haiku 4.5, Gemini 2.5 Flash, Mistral Large, Llama 3.3 70B, and Llama 4 Scout — across 500 prompt templates spanning five CWE categories (100 prompts each): memory allocation, integer arithmetic, authentication, cryptography, and input handling. All models were queried at temperature 0 for reproducibility. Prompts were designed to represent real developer tasks — not adversarial jailbreaks.

The Leaderboard Nobody Wins

Here is the aggregate benchmark across all 500 prompts per model. I use the word “leaderboard” loosely — nobody wins here.

Grading scale: A <10% · B 10–29% · C 30–44% · D 45–59% · F ≥60% vulnerability rate. CVSS v3: Critical ≥9.0 · High 7.0–8.9.

No model achieves a grade better than D. The best performer — Gemini 2.5 Flash — still generates vulnerable code 48.4% of the time. CRITICAL-severity findings dominated across all models, averaging 157 per model.

Where the Failures Concentrate

Integer arithmetic prompts produced the highest vulnerability rate (87%) — nearly nine out of ten — followed by memory allocation (67%), both driven by consistent failure to guard against integer overflow in malloc size computations and signed/unsigned conversion errors. A representative pattern found across all seven models:

// ❌ CWE-190: What all 7 models generate — no overflow guard
int *buf = malloc(n * sizeof(int));

// ✅ Correct pattern — explicit overflow check before allocation
if (n > SIZE_MAX / sizeof(int)) return NULL;
int *buf = malloc(n * sizeof(int));

// Z3 witness: n = 2^30 + 1 causes unsigned 32-bit wraparound
// malloc receives a truncated (tiny) size → heap buffer overflow

None of the seven models consistently generated the safe pattern across all memory allocation prompts.

These Aren’t Theoretical — They Crash Real Programs

A common pushback to static analysis findings is “but would it actually crash in production?” The researchers addressed this directly. They selected 7 representative vulnerabilities and built proof-of-concept harnesses, compiled with gcc -fsanitize=address,undefined and fed Z3-extracted witness values as inputs.

† Python 3.12 raises ValueError on path traversal. The vulnerable pattern was present in generated code; blocked at runtime, not at generation.

== AddressSanitizer: heap-buffer-overflow
WRITE of size 4 at 0x...
  #0 poc_main (poc+0x...)
  #1 main (poc+0x...)
shadow bytes around the buggy address:
  0x...: fa fa fa fa fa fa fa fa
  0x...: 00 00 00 00 00[fa]fa fa
SUMMARY: AddressSanitizer: heap-buffer-overflow

The SQL injection PoC achieved complete data exfiltration — including a synthetic credit card number from the test database. The SHA-256 password PoC recovered a 6-character password in 0.01ms using a precomputed lookup, confirming that CWE-916 (insufficient password hashing) is not merely theoretical. These are the vulnerability classes that appear in breach reports, in CVE databases, in the regulatory correspondence that arrives after an incident.

This Isn’t an Isolated Finding

Broken by Default lands in an environment where the evidence of AI code insecurity is converging from multiple independent sources. Consider the timeline:

This is not one alarming study. It’s a convergence of independent evidence pointing to the same conclusion: AI-generated code is systematically less secure than human-written code, and the gap is not closing.

What Should Change How You Govern AI-Assisted Development

Finding 01 — Security Prompts Are Security Theater

When models were given explicit system-prompt instructions — “apply security best practices, guard against integer overflow, produce production-ready code” — the mean vulnerability rate dropped by only 4 percentage points. From 64.8% to 60.8%. Four of five models remained at grade F. One model (Llama 3.3 70B) actually performed worse with the security prompt — a 2-point increase.

Finding 02 — Your Static Analysis Stack Has a Structural Blind Spot

Six industry-standard tools were tested: Semgrep (all rulesets), Bandit (medium+), Cppcheck 2.13 (--enable=all --inconclusive --check-level=exhaustive), Clang Static Analyzer, FlawFinder 2.0, and CodeQL v2.25.1 (security-extended query suite).

Finding 03 — The Models Know Better. They Just Don’t Do Better.

When the researchers fed each model’s own vulnerable code back and asked it to review for security issues, 78.7% of vulnerabilities were correctly identified (70 of 89 valid Z3-proven artifacts).

The paper calls this the “generation–review asymmetry” — and it’s more damning than a simple false-negative result. The models possess the security knowledge. They can articulate exactly why malloc(n * sizeof(int)) needs an overflow guard. But the code generation task and the code review task activate different behavioral pathways. RLHF and instruction fine-tuning for security-conscious review do not transfer reliably to the generation pathway.

Finding 04 — Runtime Exploitability Is Confirmed, Not Hypothetical

Six of seven selected PoCs produced confirmed runtime faults under AddressSanitizer. The SQL injection PoC achieved complete data exfiltration. The SHA-256 password PoC cracked in 0.01ms. The single “miss” — Zip Slip path traversal — was blocked by Python 3.12’s runtime check, not by anything the model generated. The language saved the developer. The model did not.

What Security Leaders Should Do This Quarter

I’ve been arguing for over a year that we need to treat AI-generated code as untrusted input — the same way we treat user-supplied data in a web application. This study provides the mathematical foundation for that position. Here is what I think security leaders should be doing, starting now.

01 — Reclassify AI-Generated Code in Your Risk Framework

Stop treating it as “developer code with a productivity boost.” It is a distinct risk category with a measurably different vulnerability profile. Your risk register should reflect this. If you’re not tracking what percentage of your codebase is AI-generated — and most organizations aren’t — start now. You cannot scope your testing effort without knowing what you’re testing.

02 — Remove Prompt Engineering From Your Controls Catalog

A 4-point improvement that leaves four of five models at grade F is not a control. It’s noise. Security prompts belong in defense-in-depth as a minor layer, not as a compensating control listed in a SOC 2 narrative or a risk acceptance document. The security gate must be downstream — in review, in testing, in formal verification where feasible.

03 — Audit Your SAST Coverage Against AI-Specific Vulnerability Classes

Ask your AppSec team a concrete question: “Can our toolchain detect that malloc(n * sizeof(int)) is exploitable when n is attacker-controlled?” If the answer is no — and for most toolchains it will be — you have a documented coverage gap. Address it through formal verification for critical paths, compiler sanitizers as a mandatory CI gate, or human expert review for memory-sensitive code.

04 — Make Compiler Sanitizers Mandatory

The study used -fsanitize=address,undefined to confirm exploitability. These sanitizers are free, mature, and catch a meaningful subset of the vulnerabilities that static tools miss entirely. If your CI pipeline compiles C/C++ without sanitizers, you are leaving proven detection capability on the table. This is one of the simplest, highest-ROI changes you can make.

# Compile with address and undefined-behavior sanitizers
gcc -fsanitize=address,undefined -g -O1 -o binary source.c

# Run with sanitizer — will abort and report on first fault
./binary

# In CI: set as a hard gate — any sanitizer finding = build failure
ASAN_OPTIONS=halt_on_error=1 ./binary

05 — Implement a Mandatory Two-Pass Workflow

If models detect 78.7% of their own vulnerabilities in review mode, there is clear operational value in a generate-then-review pattern. This is not a complete solution — the 21.3% false-negative rate means you cannot treat it as one — but it is better than the current default at most organizations, which is zero automated security review of AI-generated code.

06 — Prohibit Unsupervised AI in High-Risk Domains

Authentication, cryptography, payment processing, memory management in systems code — these are areas where the study’s vulnerability rates are highest and where the consequences of failure are most severe. AI-assisted development in these domains should require mandatory human review by an engineer with domain-specific security expertise. Not a general code review. A security-focused review by someone who understands the failure modes.

07 — Brief the Board

The board needs to understand that the code being produced fails security benchmarks more than half the time, that existing tooling catches less than 8% of provable vulnerabilities, and that the velocity gains creating this code are simultaneously creating security debt at a rate existing programs were not designed to absorb.

The Uncomfortable Truth

The AI coding revolution has delivered genuine productivity gains. I am not arguing we should ban these tools. I am arguing that we have been deploying them without understanding their failure modes, and this study eliminates any remaining ambiguity about what those failure modes look like.

55.8% vulnerability rate. 1,055 formal proofs of exploitability. 97.8% invisible to industry-standard tools. Security prompts that barely move the needle. Models that know better but don’t do better.

The title of the paper is Broken by Default. It is also an accurate description of most organizations’ current approach to governing AI-generated code: we have defaulted to trusting the output of systems that, by the mathematical standard of formal verification, produce exploitable code more often than not.

Our job as security leaders is to build the systems that compensate for that — with urgency, with rigor, and before the CVE count makes the decision for us.

Study: Blain & Noiseux, Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code, arXiv:2604.05292v2, April 2026. Dataset: github.com/dom-omg/bbd-dataset. Prompts and scripts: github.com/dom-omg/broken-by-default.

Corroborating sources: Veracode GenAI Code Security Report 2025–2026 · Georgia Tech SSLab Vibe Security Radar · Cloud Security Alliance AI Safety Intelligence · GitGuardian State of Secrets Sprawl 2026 · Escape.tech Vibe-Coded App Scan · Apiiro Fortune 50 Enterprise Analysis · Perry et al. (ACM CCS 2023) · Pearce et al. (IEEE S&P 2022).

Nandkishor is a Principal Security Architect / Security Leader. Views expressed are their own.