Bug Hunt Bench grades frontier coding models on 105 real bugs in production repos
PawelHuryn · x · 2026-09-13
Developer Pawel Huryn released Bug Hunt Bench: 105 real bugs planted in two production codebases, with frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM and more) getting one round per repo in their own agentic CLIs (Codex CLI, Claude Code, Grok CLI, Antigravity CLI). Every diff is blind-graded against a withheld answer key, counting only planted bugs. Live leaderboard includes method notes and every receipt. Early finding: Muse Spark 1.3's xhigh mode is 50% pricier and 13% slower than high but fixes just one more bug — max effort seems to buy the biggest single jump.
Related event: Bug Hunt Bench: Muse Spark 1.3 Ties Fable 5.1 for the Top Spot(6 posts)→
More from coding & agent
- GitHub Copilot app is building richer dashboard widgets for personalized data views — mariorod1 · 2026-10-31
- Todoist's self-healing agent auto-fixes 1 in 5 Sentry issues and opens PRs — zeeg · 2026-10-31
- Dev's Multica 1.0 build log: 432M tokens, 89 agent runs, just $183 — jiayuan_jy · 2026-10-31
- Stardock's QA now runs agents that play the game and file reports — draginol · 2026-10-31
- Limma MCP structures unstructured data end-to-end so agents never miss a detail — FlimsyIngenuity7248 · 2026-10-31
- Agent audit metric: can you name which tests ran and files changed without reading its summary? — ronbodkin · 2026-10-31