Bug Hunt Bench Ranks Frontier Coding Models on 105 Real Bugs with 200x Cost Gap
Pawel Huryn launched Bug Hunt Bench, a blind-graded leaderboard testing frontier coding models on 105 deliberately planted bugs across two repos, revealing a 200x cost gap between models; the author says his code-review skill significantly boosts model scores.
2026-09-24 ~ 2026-09-24 · 3 related posts
- Episode 1: OpenAI Launches GPT-6 Sol and Luna at Half Price; Big Hallucination Gains but Not a Clean Sweep(2026-09-23, 92 posts)
- Episode 2: GPT-6 Sol/Luna launch splits opinion: half-price but dumber?(2026-09-23, 9 posts)
- Episode 3: LMArena Opens GPT-6 Sol and Luna for Public Testing(2026-09-23, 2 posts)
- Episode 4: Four New Models Reshape the Cost-Performance Frontier, Opus 5.5 Tops Intelligence Index(2026-09-23, 2 posts)
- Episode 5: OpenAI and Anthropic Slash Prices Within 90 Minutes, Kicking Off a New Model Price War(2026-09-23, 7 posts)
- Episode 6: Tests Show GPT-6 Sol Scales Nearly Linearly With Effort(2026-09-23, 3 posts)
- Episode 7: GPT-6 Astra Tops Bug Hunt Bench While Sol Version Regresses(2026-09-24, 2 posts)
- Episode 8: Leaks Map GPT-6 Codenames to GPT-5.6 Versions(2026-09-24, 3 posts)
- Episode 9: Bug Hunt Bench Ranks Frontier Coding Models on 105 Real Bugs with 200x Cost Gap(2026-09-24, 3 posts)
- Episode 10: GPT-6 Sol and Luna Launch With Aggressive API Pricing(2026-09-25, 2 posts)
- Episode 11: OpenAI's efficient GPT-6 models seen as freeing compute for bigger frontier models(2026-09-25, 2 posts)
- Episode 12: GPT-6 Sol Underestimated: 1.6x Robot Task Gains at 47% Lower Cost(2026-09-25, 2 posts)
- Episode 13: GPT-6.1 Sol Launches with Astra-Level Intelligence at a Fifth of the Price(2026-09-29, 32 posts)
- Episode 14: GPT-6 Astra Ultrafast Reportedly 8x Faster in Codex(2026-09-30, 2 posts)
- Episode 15: GPT-6.1 Sol third-party reviews: near-Astra performance at a fraction of the cost(2026-09-30, 21 posts)
- Bug Hunt Bench author says his code-review skill significantly lifts model scores — PawelHuryn · 2026-09-24
- Bug Hunt Bench leaderboard live: 105 planted bugs, score vs cost across frontier models — PawelHuryn · 2026-09-24
- Bug Hunt Bench: frontier coding models graded on 105 planted real-world bugs, cost spread ~200x — PawelHuryn · 2026-09-24