Bug Hunt stats: Fable 5.2 found just 1 never-before-fixed bug across 300+ runs
PawelHuryn · x · 2026-11-10
Pawel Huryn shared anonymous coverage stats from his Bug Hunt benchmark (answer keys stay private): across 300+ runs, only one bug fixed by Fable 5.2 (max) had never been fixed before by any model. An accompanying heatmap shows per-model coverage over 105 planted bugs:
- The benchmark uses real repos and re-planted bugs that frontier models missed in early 2026; only fully fixed bugs count, scored by blind calibrated judges from a different model family.
- Highlights: Sonnet 5.5 (max) 51/105, GPT-6 Astra (max) 45/105, DeepSeek V4.1 Flash 1017 (max) 42/105; each model's fixed-bug set differs substantially — per run, 15 bugs on average are fixed by only that model.
- Some models (e.g. certain OpenAI ones) report many real but irrelevant issues to game the benchmark; these aren't counted.
Related event: Bug Hunt Bench: Fable 5.2 Tops Ranking, Fixing 58/105 Real Bugs(3 posts)→
More from Models
- Musk: Grok 4.8 ditches JAX/SGLang for from-scratch C/C++ stack, now in RL — tetsuoai · 2026-11-10
- Same prompt head-to-head: Fable 5.2 costs 76% more than Opus 5.5 but finishes 1h15m faster — FinanceYF5 · 2026-11-10
- Qwen3.6-35B on one RTX 3060 12GB: RustingStrata beats llama.cpp in new benchmark — Serious_Scarcity7818 · 2026-11-10
- Fable 5.2 system prompt leaked: 420,000+ characters, full text on GitHub — morgymcg · 2026-11-10
- ChatGPT remote control on Android plagued by failures, users say Claude works better — koltregaskes · 2026-11-10
- Screenshot shows Gemini explaining 'load shedding' of users to cheaper models — Top-Sink-1315 · 2026-11-10