Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value
Independent evaluator Pawel Huryn (author of The Product Compass) built and published Bug Hunt Bench: 105 bugs drawn from two real code repositories, with frontier models asked to find and fix them, scored by blind judges from different model families against a private answer key. GPT-6 Astra (max) topped the board fixing 45 ($33.03, n=3); Opus 5.5 matched GPT-6 at roughly half the cost; Xiaomi's newly released MiMo-V2.6-Pro scored 22.7 for just $0.86, which Huryn called the value benchmark; Grok 4.7 showed no meaningful gain over 4.6.
Confirmed
- Method: two real repos, one prompt per repo, blind cross-family judging against an answer key, unfixed/missed bugs scored 0, multiple runs per model with statistics reported, published at Bug Hunt Bench (bughunt.productcompa…).
- Results: GPT-6 Astra (max) 45, first ($33.03, n=3); Opus 5.5 ties GPT-6 at about half the cost; Muse Spark 1.3 (max) 32.2 ($18.11); GPT-5.6 Luna (max) 31.5 ($2.82); MiMo-V2.6-Pro 22.7 ($0.86); Grok 4.7 and 4.6 both 28.7. Costs differ by nearly 200x.
- Interim leaderboard (m8, m10): GPT-6 Astra (max) 45 (n=3), Fable 5.1 (max) 43 (n=1), GPT-5.6 (max) 42.5, Opus 5.5 (max) 41.7; GPT-6 Sol (max) fixed only 29.3, a sharp regression to last place (m1, m9); GPT-5.6 Sol (max) 43.5. Huryn also noted Opus 5.5 loses more score at lower reasoning effort.
- Difficulty clarification (m14): the 105 bugs were not randomly sampled but hard problems frontier models initially failed in early 2026; real bugs not deliberately planted are excluded because OpenAI would flood results with real, theoretical bugs.
- Grok follow-up (m11, m15): at high reasoning, Grok 4.6 beat 4.7 by 23 to 19, and 4.6's medium tier was also better; at max, a fourth run (n=4) put 4.7 ahead by only 0.1, judged random noise, and testing at that tier stopped; xhigh showed no separation in three runs.
Unconfirmed
- Huryn's suspicion that Grok 4.7 was optimized for popular benchmarks rather than real coding skill remains personal speculation without corroboration.
- Whether this two-repo, hard-bug-heavy benchmark generalizes to real engineering awaits independent replication; praise of MiMo's value by reposters like GregGL87 is secondhand commentary.
Why it matters
- By deliberately selecting problems all frontier models initially failed and publishing costs, the benchmark vividly exposes huge capability and price gaps (MiMo-V2.6-Pro got roughly half of GPT-6's score at 1/38 the price), giving developers a third-party reference distinct from official marketing.
- Grok 4.7 shows no real generational gain here and GPT-6 Sol regresses sharply versus Astra — valuable independent signals for anyone tracking frontier model progress.
2026-09-22 ~ 2026-09-23 · 18 related posts
- Episode 1: Grok 4.7 Briefly Surfaces in OpenCode Zen Directory, Fueling Launch Rumors(2026-09-21, 5 posts)
- Episode 2: xAI launches Grok 4.7 with major benchmark gains at unchanged price and speed(2026-09-21, 59 posts)
- Episode 3: Grok 4.7 Day-One Hands-On: Strong Word-of-Mouth for Autonomous Coding(2026-09-22, 17 posts)
- Episode 4: Grok 4.7 Launches: Big Benchmark Gains but Doubled Token Use Sparks Cost Debate(2026-09-22, 25 posts)
- Episode 5: Grok 4.7 Reportedly Closes Gap With Top Frontier Models at Lower Cost(2026-09-22, 2 posts)
- Episode 6: Open-Source Grok 4.7 Editor Extension Passes 140K Installs(2026-09-22, 3 posts)
- Episode 7: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(2026-09-22, 18 posts)
- Episode 8: Grok 4.7 Meets Xiaomi MiMo-V2.6 in a Same-Night Price War(2026-09-22, 5 posts)
- Episode 9: xAI Ships Three Grok Frontier Models in Nine Weeks(2026-09-22, 2 posts)
Primary sources
- Bug Hunt Bench ranks GPT-6 Astra top as coding models fix real planted bugs, costs spread 200x — PawelHuryn ·
- Opus 5.5 fixes 43 of 105 planted bugs, nearly matching GPT-6 Astra at double the cost — PawelHuryn ·
- Frontier-Model Bug Benchmark: 105 Problems Models Failed at the Start of 2026 — PawelHuryn ·
- 105 Planted Bugs Put Grok 4.7 at 28.7 vs GPT-6 Astra's 45 in Real-Repo Coding Test — PawelHuryn · 2026-09-22
- Reshared: Grok 4.7 matches Grok 4.6 at 28.7 on 105-bug real-world test — eugeniyoz · 2026-09-22
- Grok 4.7 beats 4.6 by just 0.1 across runs: 'random noise,' independent eval finds — PawelHuryn · 2026-09-22
- Independent Tests Show Grok 4.6 (high) Beating 4.7, 23 vs. 19 — PawelHuryn · 2026-09-22
- Xiaomi MiMo-V2.6-Pro fixes 22.7% of 105 real bugs for just $0.86 in blind test — PawelHuryn · 2026-09-22
- Bug Hunt Bench tests frontier coding models on 105 real bugs, costs vary 200x — PawelHuryn · 2026-09-22
- 105 real bugs blind benchmark: MiMo-V2.6-Pro scores 22.7 for $0.86, a strong Pareto frontier — Greg_GL_87 · 2026-09-22
- [source] Frontier-Model Bug Benchmark: 105 Problems Models Failed at the Start of 2026 — PawelHuryn · 2026-09-23
- [source] Opus 5.5 fixes 43 of 105 planted bugs, nearly matching GPT-6 Astra at double the cost — PawelHuryn · 2026-09-23
- [source] Bug Hunt Bench ranks GPT-6 Astra top as coding models fix real planted bugs, costs spread 200x — PawelHuryn · 2026-09-23
- GPT-6 Astra tops ongoing eval at 45; Opus 5.5 drops hard at lower effort levels — PawelHuryn · 2026-09-23
- Community evals put GPT-6 Astra (max) at 45, ahead of Fable 5.1 and Opus 5.5 — PawelHuryn · 2026-09-23
- GPT-6 Sol Tanks Hands-on Bug Hunt: Fixes 29 of 105 Hidden Bugs, Cheapest at $9.93 — PawelHuryn · 2026-09-23
5 near-duplicate retellings: PawelHuryn · PawelHuryn · PawelHuryn · PawelHuryn · ronnhue