Nearly 60% of AI-written formal specs proved the wrong thing, study finds
burny_tech · x · 2026-10-26
A recent study put a number on "vibe verification": AI wrote the spec, AI wrote the proof, and the verifier accepted both — but when humans read the specs, nearly 60% proved something other than what the author meant.
The proofs weren't wrong — the specs were. Left to choose what to prove, agents pick properties that are easy to prove and say nothing meaningful about the program, and the checker can't distinguish a useless theorem from a useful one.
Quoting @derekcheng: this exposes a fundamental paradox of formal verification with LLMs — if you trust an LLM to flawlessly convert your legacy codebase into a formal spec, wouldn't you just trust its initial code output anyway?
More from coding & agent
- Agent payments are already live via x402 — but what happens when a transaction stalls mid-payment? — SpreadUsual4084 · 2026-10-26
- Storyboard grids tame MiniMax reference-to-video for complex music video scenes — Cheap_Credit_3957 · 2026-10-26
- Connecting Copilot Studio to SAP S/4HANA via MCP with per-user identity — ColdImportance2484 · 2026-10-26
- Base44 fine-tuned Qwen 2B in half a day for prompt autocomplete serving millions — tristanbob · 2026-10-26
- Grok highlights Blender MCP demos: FIFA-style playable game from a single prompt — chongdashu · 2026-10-26
- Ex-frontier-lab contractor launches free AI engineering curriculum: 208 handbooks, 130 interactive builds — iamsausi · 2026-10-26