Bug Hunt Bench: Gemini 4 Carbon scores 57.3 for $31, nearly matching Fable 5.2 at $140
PawelHuryn · x · 2026-11-11
Testing models on 105 planted bugs across 2 repos with Bug Hunt Bench, the author finds Gemini 4 Carbon (high) scored 57.3 for just $31.32, just behind Fable 5.2 max (58, $139.65). Sonnet 5.5 max hit 51.3 for $154.03; GPT-6 Astra 45 for $33.03; GPT-6.1 Sol 44.3 for $5.88; Gemini 4 Argon 40.7 for $7.16. The takeaway: Google's benchmarks weren't hype—Carbon is +16.6 points over Argon at less than a quarter of Fable 5.2's cost. The attached leaderboard also lists DeepSeek V4.1 Flash 1017 (42.7, $1.84) and Kimi K3.1 Pro (36), with wall-clock times and cost per run.
Related event: Bug Hunt Bench: Fable 5.2 tops 105 real-bug leaderboard(6 posts)→
More from coding & agent
- Gemini 4 Carbon scores 57.3 fixing real bugs for $31, just behind Fable 5.2's 58 at $140 — PawelHuryn · 2026-11-11
- Supabase lands in Expo EAS CLI: one command wires a backend into every environment — dshukertjr · 2026-11-11
- What would you rewrite with AI? Orosz sparks a list of cheaper rebuilds of existing services — _philschmid · 2026-11-11
- Ex-Meta researcher predicts a hard ceiling of 150-200B tokens for the SWE market — DimitrisPapail · 2026-11-11
- ComfyUI Challenge Winner ComfyArt: Open-Source Fully Local AI Design Canvas — bfl_ai · 2026-11-11
- Letting an agent self-improve its own harness daily: resumed runs 7→77, memory noise 94%→0 — AccBalanced · 2026-11-11