Gemini 4 Carbon at medium matches high tier on Bug Hunt bench, 20% cheaper and 29% faster
PawelHuryn · x · 2026-11-11
Pawel Huryn published fresh Bug Hunt Bench results (105 real bugs frontier models missed in early 2026; only fully fixed, pre-planted bugs count):
- Gemini 4 Carbon high: 57.3 for $31.32, 114 min
- Gemini 4 Carbon medium: 56 for $24.95, 81 min
- Medium is level with high, costing 20% less and running 29% faster; high's runs spread 52–62 while medium stayed 55–57
- Both beat everything else on the board; only Fable 5.2 (max) leads, by 0.7 and 2 points
- Carbon (medium) beats Fable 5.2 (xhigh) 56 vs 49 at 41% of the cost
Methodology notes: "found but almost fixed" scores 0; unplanted bugs aren't counted because OpenAI models report irrelevant "bugs" to game the benchmark; Sonnet 5.5 (max)'s 51.3 is real and comes from making 7x more turns than GPT-6 Astra (max).
More from coding & agent
- Agent Verification Without a Deadline Is a Log Entry, Not a Safeguard — Key-Register-1969 · 2026-11-11
- Free open-source course: build a Claude Code clone from scratch across 8 articles and 6 videos — blaizedsouza · 2026-11-11
- Baidu open-sources ERNIE 5.5 Flash Preview, which coded a wuxia game for 11 hours straight — xiaohu · 2026-11-11
- mcp-slim: Zero-Config MCP Proxy Cuts Tool Schema/Output Tokens 50–80%, Redacts Secrets — Leading_Strain3682 · 2026-11-11
- Currents Ships a CI-Testing MCP Server: 43 Tools, Split OAuth Resources, Skills over MCP — sumitsaurabh121 · 2026-11-11
- You can get 2.5M free GPT tokens a day via OpenAI's data-sharing program — crasspuk · 2026-11-11