Bug Hunt Bench tests frontier coding models on 105 real bugs, costs vary 200x
PawelHuryn · x · 2026-09-22
Pawel Huryn launched Bug Hunt Bench, grading frontier coding models on 105 real (non-planted) bugs from Xiaomi repos with blind cross-family judges against a secret answer key. Results (score/cost): Muse Spark 1.3 max 32.2/$18.11; GPT-5.6 Luna max 31.5/$2.82; Grok 4.7 xhigh 28.8/$22.89; MiMo-V2.6-Pro 22.7/$0.86; DeepSeek V4.1 Flash 21.7/$0.78; Gemini 3.8 Flash 18/$11.03. Costs span 200x, forming a strong Pareto frontier. Full notes and data are on the benchmark site and GitHub.
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from coding & agent
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Free AI game jam in SF Nov 14–15 with teams from Anthropic, Google, Meshy, OpenAI — chrisfirst · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Coding harnesses hand shell access to inference providers — 4-step gateway mitigation — shensi · 2026-11-12
- Slack's David Hoang: agents beat people on coordination, proposes 'boundary agents' — davidhoang · 2026-11-12