Bug Hunt Bench: Grok 4.7 (medium) consistently beats its xhigh reasoning tier on planted-bug tests
PawelHuryn · x · 2026-09-22
Pawel Huryn's Bug Hunt Bench tests frontier coding models against 105 real planted bugs with blind grading, and costs across runs span roughly 200x.
Early findings:
- Grok 4.7 (medium) consistently beats (high): individual runs were 30/23/27 for medium vs 19/22/17 for high, while xhigh scored 25/30/31/29 and low only 11/15.
- Higher reasoning effort doesn't translate into better bug-fixing, questioning the cost-performance tradeoff.
- Muse Spark 1.3 shows no clear progression either.
The board bills itself as hands-on, no hype, with full run notes and caveats published on GitHub.
Related event: Independent tests show Grok 4.7 medium reasoning beats xhigh(2 posts)→
More from coding & agent
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- Dev's favorite AI use case: Claude monitors competitors and emails alerts — doooyle · 2026-11-12
- LLM controls vocal synth to perform Ave Maria in 4-part harmony — AAAzzam · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Claude watches competitors around the clock and emails me what matters — GlenBradley · 2026-11-12
- Developers say Gemini 4 Carbon is notably better at favicon design and micro-interactions — AI_Andrew · 2026-11-12