Gemini 4 Carbon scores 57.3 fixing real bugs for $31, just behind Fable 5.2's 58 at $140
PawelHuryn · x · 2026-11-11
The author blind-tested frontier models on 105 replanted real bugs across two real repos, counting only genuinely fixed bugs:
- Results & cost: Fable 5.2 (max) scored 58 for $139.65; Gemini 4 Carbon (high) scored 57.3 for just $31.32; Sonnet 5.5 (max) hit 51.3 for $154.03; GPT-6 Astra (max) 45 for $33.03; GPT-6.1 Sol (max) 44.3 for only $5.88.
- Method: blind judges from different model families score against a private answer key; unplanted bugs don't count, preventing models from gaming the benchmark by reporting irrelevant "bugs".
- Details: Carbon's score is the mean of 3 runs (58/62/52); Sonnet 5.5 used 7x more turns than GPT-6 Astra; the API's xhigh/max effort levels silently fall back to default.
More from coding & agent
- Fable 5.2 tested: max effort costs 25x more than medium for just +0.06 score — aakashgupta · 2026-11-11
- Supabase lands in Expo EAS CLI: one command wires a backend into every environment — dshukertjr · 2026-11-11
- What would you rewrite with AI? Orosz sparks a list of cheaper rebuilds of existing services — _philschmid · 2026-11-11
- Ex-Meta researcher predicts a hard ceiling of 150-200B tokens for the SWE market — DimitrisPapail · 2026-11-11
- ComfyUI Challenge Winner ComfyArt: Open-Source Fully Local AI Design Canvas — bfl_ai · 2026-11-11
- Letting an agent self-improve its own harness daily: resumed runs 7→77, memory noise 94%→0 — AccBalanced · 2026-11-11