'Vibe verification': ~60% of AI-written formal specs prove the wrong property
basedjensen · x · 2026-10-26
Quoting a paper cited by Alex Lavaee: when AI writes both the formal spec and the proof, the verifier accepts everything — but humans reading the specs find nearly 60% prove something other than what the author meant. Not wrong proofs; wrong specs.
The verifier is right every time, and that's the problem: left to choose, the agent picks properties that are easy to prove but say nothing about the program, and the checker can't distinguish a useless theorem from a useful one. He coins the term vibe verification: the agent writes spec and proof, the checker goes green, nobody reads the spec.
Separately, Doug (0xdoug) mocks a paper claiming LLMs can't write specs: its test models turned out to be GPT-4o and Llama — neither a modern RLVR-trained model nor a reasoning model.
More from coding & agent
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Prompt turns Claude Code into a forensics tool finding deleted iPhone photos in backups — alex_verem · 2026-10-27
- Phones won't be mobile IDEs but control planes for agent fleets, argues Yuchen Jin — Yuchenj_UW · 2026-10-27
- FrontierCS 2 launches on Kaggle: 48 tasks make agents beat published CS papers — MeganRisdal · 2026-10-27
- Stop remaking old games with AI: a call for more ambitious agentic software — rschu · 2026-10-27