Bug Hunt Bench: Qwen3.8-27B run 3x beats single Opus 4.8 max on planted bug detection
PawelHuryn · x · 2026-09-17
Pawel Huryn launched Bug Hunt Bench, blind-testing frontier coding models on 105 real repos with planted bugs. Key findings:
- The bugs are tough ones frontier models missed at the start of 2026, not random trivia — yet Qwen reviewing a normal PR catches far more
- Running Qwen3.8-27B (8-bit) three times beats a single run of Opus 4.8 (max) in detection coverage, showing the payoff of repeated sampling for bug hunting
- Costs across the leaderboard span roughly 200x (log scale); runtime under 7x
The board is blind-graded with one prompt per repo; data and caveats are public on GitHub, and only runs the author executed are listed.
Related event: Bug Hunt Bench: local Qwen3.8-27B nears Claude Opus in bug fixing(3 posts)→
More from Models
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Yacine calls Terra High 'kind of an insane model' — yacineMTB · 2026-11-02
- Running DeepSeek V4.1 on a single H200 at $5.30/hr — daluoseo · 2026-11-02
- Leaked Fable 5.5 checkpoint video called 'pure code' — 'will change the game' — kimmonismus · 2026-11-02
- Claude Opus 5.5 quality complaints spark debate on why continual learning is so hard — JeremyNguyenPhD · 2026-11-02
- Why is AI sexist? Reddit debate on models overlooking Iceland's women's strike — Mathemodel · 2026-11-02