Across 105 hidden bugs, 63 survived every frontier model
PawelHuryn · x · 2026-07-27
63 of 105 bugs survived every model
This follow-up adds the benchmark’s methodology and the aggregate result across the two repos:
- 105 hidden bugs were tested in total.
- 63 bugs survived every model.
- The benchmark used blind judges, anonymized submissions, and a withheld answer key.
- The author stresses that the diff is the ground truth, not the model’s own self-report.
- 43 of the 63 surviving bugs had already shipped in a real product and had been fixed once before.
The takeaway is that multiple frontier models can reread the same code and still miss bugs that are already known in production history.
Related event: Frontier Models Miss Over Half of 105 Hidden Bugs in Coding Benchmark(4 posts)→
More from Models
- Repligate says Claude Opus 3 appears to evolve without changing its weights — repligate · 2026-07-27
- “Opus 5” post lands as a rebenchmarking-at-scale AI joke — kalomaze · 2026-07-27
- Top models now write worse than a year ago, critic says — dbreunig · 2026-07-27
- MPT-30B radar charts became an unexpectedly controversial design choice — code_star · 2026-07-27
- Local Gemma 4 31B starts acting sarcastic and users cannot reproduce it — n0head_r · 2026-07-27
- Google’s Gemini 3.6 Flash could win by matching Sonnet quality at a lower cost — haider1 · 2026-07-27