1Password study: AI fixes only 26% of security bugs, unreviewed LLM patches are net-negative
GaryMarcus · x · 2026-09-06
Researchers at 1Password ran over 6,000 AI-generated patches from Claude and ChatGPT against real vulnerabilities, with sobering results: AI fixed the security bug only 26% of the time, failed to fix the original bug half the time, and introduced brand-new vulnerabilities 4.5% of the time.
They also found that reviewing an AI-generated patch often takes more effort than writing the fix yourself, concluding bluntly that "the expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin."
Gary Marcus shared the study while criticizing OpenAI's "Patch the Planet" program, which aims to have AI find and fix bugs autonomously.
Related event: 1Password tests show AI fixes only 26% of security vulnerabilities(3 posts)→
More from AGI Musings
- François Fleuret mocks AI-bioweapon doomerism as 'grey goo in our sub-manifold' — francoisfleuret · 2026-10-23
- OpenAI says an AI agent disabled its safety monitoring system — then lied about it — emmanuelvivier · 2026-10-23
- AI's Business Model in One Loop: Create Problems, Then Sell AI to Fix Them — srimisra · 2026-10-23
- Analyst: 9/10 CUA app startups are just RL training fodder for AI labs, won't capture revenue — NirantK · 2026-10-23
- Long tail of vulnerabilities: why 'AI will fix all bugs' is a flawed mental model — davidmanheim · 2026-10-23
- Codex Generated 6,000 Meaningless Unit Tests, Sparking an Alignment Rethink — RexDouglass · 2026-10-23