Frontier AI models fix only 1 in 4 security vulnerabilities correctly, report finds
Evgenii42 · reddit · 2026-09-06
A Reddit post highlights a report from security researchers at Off-by-1 Labs (published via 1Password) testing frontier LLMs on patching known non-trivial security vulnerabilities:
- Success rate is only 25%; in other cases models failed to fully patch the issue, made unreviewed changes, or introduced new vulnerabilities;
- Bottom line: current frontier models can't be trusted to autonomously fix security bugs;
- Human review is impractical: properly reviewing AI patches demands such high cognitive effort that writing the patch manually is easier;
- The report includes practical guidance on what context to provide LLMs for better patches — worth reading if you use AI for coding.
Related event: 1Password tests show AI fixes only 26% of security vulnerabilities(3 posts)→
More from Models
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- TypeSafe's Jev: a "Decision Model" LLM That Returns Floats at $0.042/M Input Tokens — ariG23498 · 2026-10-23
- Microsoft's MAI-Base-2 scaling ladder pegs the L42 run at ~1.8×10^23 FLOPs — teortaxesTex · 2026-10-23
- Local model agentic coding benchmark: 27B matches 125B, tool-calling is the lever — Joralk_ · 2026-10-23
- NVIDIA open-sources PersonaPlex-7B, a full-duplex voice AI model that listens and talks simultaneously — solyarisoftware · 2026-10-23
- NVIDIA's KumoRFM-2: a foundation model that predicts churn, fraud and demand straight from relational databases — solyarisoftware · 2026-10-23