105 real bugs benchmarked: Opus 5.5 and Muse Wonder lead, DeepSeek V4.1 Flash 1017 jumps to 34
PawelHuryn · x · 2026-10-17
Pawel Huryn planted 105 hidden bugs in two real repos to test frontier coding models on his Bug Hunt Bench:
- Leaderboard: Muse Wonder Full 1.0 (max) tops at 42, Opus 5.5 (max) 41.7, Gemini 4 Argon (high) 40.7. DeepSeek V4.1 Flash 1017 (max) scores 34 — a big jump from 21.7 for the older snapshot. Grok 4.7 (xhigh) 29, Qwen3.8-Max 25.7, GLM-5.3 19.
- FAQ: all are real bugs frontier models initially missed in early 2026; Sonnet 5.5 (max) earns its score by spending 7x more turns than GPT-6 Astra (max); unplanted bugs don't count — some OpenAI models report many real but irrelevant issues to game the score.
- Design: blind-graded, one prompt per repo, sortable by score/cost/time/turns, full data on GitHub.
Related event: Open-Source 763B Model Tested on 105 Hidden Bugs(3 posts)→
More from coding & agent
- Having your agent read 8,000-word articles may be the future of serious online writing — danshipper · 2026-10-17
- Tencent open-sources Hy-Memory 2.0 for cross-tool agent memory sharing — aigclink · 2026-10-17
- Approval flows should follow the risk, not the org chart — usehive · 2026-10-17
- OpenDesign Motion: An Open-Source Alternative to Claude Motion for Video — petewoodbridge · 2026-10-17
- Durable PTY: Open-Source Tool Keeps AI Coding Agents Alive Across Crashes and Reboots — jasonkneen · 2026-10-17
- Deer forest demo vibe-coded in 1 hour with Tripo API and Claude — Scobleizer · 2026-10-17