Trail of Bits: 1Password's '26% clean fixes' AI patching benchmark is misleading
austinc3301 · x · 2026-09-15
Trail of Bits systematically rebuts 1Password's August 2026 report claiming AI models produce clean security fixes only 26% of the time, calling the headline figure misleading.
Key criticisms:
- Cherry-picked hard bugs: the six vulnerabilities were chosen for complex fixes; per-bug clean-fix rates ranged from 3% to 60%, so the average depends heavily on sample selection.
- 22% of trials were told to fail: two prompts explicitly instructed agents to apply the wrong fix, yet counted toward the average.
- Over a third couldn't test: one eval mode (36% of data) blocked agents from compiling or running code.
- Models under-tested: GPT-5.5 at medium effort and Opus 4.8 at high, neither at the maximum setting.
The authors warn defenders may abandon repairable vulnerabilities if they take the headline at face value. They also release real patch-quality data from their consulting work and Patch the Planet, plus two open-source agent skills: post-patch-validation and review-walkthrough.
More from Models
- Bug Hunt Bench: multiple runs boost small-model bug detection but move frontier models just 1-2 points — PawelHuryn · 2026-09-15
- ZDTaichu5.0-9B, a 9B vision-language model with spatial reasoning, trends on Hugging Face — TaichuAI · 2026-09-15
- Which 10Eros video-model quant works best on 8GB VRAM? A practical trade-off question — apostrophefee · 2026-09-15
- rasbt shows why final-result benchmarks mislead: Astra vs Qwen in Paint — rasbt · 2026-09-15
- Cristóbal Valenzuela praises Solaris: 'Websites are going to be fun again' — c_valenzuelab · 2026-09-15
- Google DeepMind on speech-to-speech: conversational, intelligent, multimodal — pick two — AI Engineer · 2026-09-15