1Password tested 6,000 AI-generated security patches: only 26% actually fixed the bug
AccBalanced · x · 2026-09-07
Researchers at 1Password ran over 6,000 AI-generated patches — built with Claude and ChatGPT — against real vulnerabilities. AI fixed the bug only about 26% of the time, failed to fix the original bug half the time, and introduced a brand-new vulnerability 4.5% of the time.
Worse, reviewing an AI-generated security patch took more effort than just writing the fix yourself. Their blunt conclusion: "The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin."
The post also notes OpenAI's summer "Patch the Planet" program, where AI finds bugs and generates fixes — a thesis this study directly undermines.
Related event: 1Password tests show AI fixes only 26% of security vulnerabilities(3 posts)→
More from Models
- Diffusion LLM bias rankings shift with sampling configs, persist after debiasing — Bollegala · 2026-10-23
- Running ~100 DeepSeek v4.1 flash agents in parallel for under $20/month — zack_overflow · 2026-10-23
- Researcher laments open-source audio models are surprisingly subpar, even representation ones — gowthami_s · 2026-10-23
- User claims they generated video with "Claude Opus 5.5", unannounced model name surfaces — jwt0625 · 2026-10-23
- Security researchers push back as OpenAI restricts Astra access; BlackHat 2027 mocked as 'cyber model' bazaar — cyb3rops · 2026-10-23
- Mistral Large 4 Leaked on Hugging Face: 1T Params, 49B Active, Open Weights Oct 31 — mervenoyann · 2026-10-23