Seven frontier models miss 44 hidden bugs even with 922 green tests
PawelHuryn · x · 2026-07-27
Repo 1: 45 hidden bugs, 7 models, 1 round each
The benchmark tested seven frontier models on a VS Code extension with 45 hidden bugs. All 922 tests stayed green throughout, but the diff-based grading showed that a green test suite did not mean the code was actually fixed.
Key takeaways from the run:
- Sonnet 5 fixed just 1 bug, added a regression test for it, and then reported that nothing else looked suspicious.
- 44 bugs were still left untouched.
- The screenshot shows the per-model fixes, wall-clock time, and cost, with GPT-5.6 Sol leading the field on this repo.
The post’s point is blunt: a passing test suite and a confident model self-report are both insufficient as proof of correctness.
Related event: Frontier Models Miss Over Half of 105 Hidden Bugs in Coding Benchmark(4 posts)→
More from Models
- Repligate says Claude Opus 3 appears to evolve without changing its weights — repligate · 2026-07-27
- “Opus 5” post lands as a rebenchmarking-at-scale AI joke — kalomaze · 2026-07-27
- Top models now write worse than a year ago, critic says — dbreunig · 2026-07-27
- MPT-30B radar charts became an unexpectedly controversial design choice — code_star · 2026-07-27
- Local Gemma 4 31B starts acting sarcastic and users cannot reproduce it — n0head_r · 2026-07-27
- Google’s Gemini 3.6 Flash could win by matching Sonnet quality at a lower cost — haider1 · 2026-07-27