Bug Hunt Bench leaderboard ranks coding models on planted-bug fixes, cost and runtime
PawelHuryn · x · 2026-11-13
AI researcher Pawel Huryn shared results from Bug Hunt Bench, which scores coding agents only on fixing planted bugs (unplanned fixes counted separately):
- Claude (Fable 5.2 Max) leads at 58/105 with n=3, costing $139.65 and 142.8 min
- Gemini 4 Carbon High follows at 57.3 for just $31.32
- Sonnet 5.5 Max scored 51.3 but took 2871 min and $154
- OpenAI's GPT-6 Astra/Sol family landed at 43-45; GPT-6.1 Sol Max cost only $5.88
- DeepSeek V4.1 Flash 1017 hit 42.7 at just $1.84; Kimi K3.1 Pro scored 36, Grok 4.7 29 (noted as a reconstructed lower bound since its CLI reports token occupancy, not billing)
Individual xhigh-effort runs showed high variance: 43, 27, 28.
More from coding & agent
- Open-source getmcp-cli turns 1,280-endpoint OpenAPI specs into 25-tool MCP servers, cutting tool tokens from 263k to 6k — Worried-Barracuda845 · 2026-11-13
- Formal verification is the Rust of software correctness — LLMs are making people write specs — arpit_bhayani · 2026-11-13
- whats-in-my-claude: open-source local visualizer for Claude Code's .claude folder, runs fully offline via npx — Srqi · 2026-11-13
- Delulu Bio adds an MCP server to its free link-in-bio tool, letting Claude manage your page by voice — delulu_intern · 2026-11-13
- Custom motion-director skill turns any voiceover into 1080x1920 motion-graphics animations with Gemini Spark — mijozakinho_2 · 2026-11-13
- Redditor builds an entire programming language, Math-Engine-MEP, entirely with Gemini — Time_Warp_Anomaly258 · 2026-11-13