1Password tests show AI fixes only 26% of security vulnerabilities

A joint report by Off-by-1 Labs and 1Password tested over 6,000 patches generated by Claude and ChatGPT for real vulnerabilities, finding only about 26% actually fixed the bugs, warning against trusting AI fixes without review.

2026-09-06 ~ 2026-09-07 · 3 related posts

• Frontier AI models fix only 1 in 4 security vulnerabilities correctly, report finds — Evgenii42 · 2026-09-06
• 1Password study: AI fixes only 26% of security bugs, unreviewed LLM patches are net-negative — GaryMarcus · 2026-09-06

1 near-duplicate retellings: AccBalanced