Bug Hunt Bench: Qwen3.8-27B run 3x beats single Opus 4.8 max on planted bug detection
PawelHuryn · x · 2026-09-17
Pawel Huryn launched Bug Hunt Bench, blind-testing frontier coding models on 105 real repos with planted bugs. Key findings:
- The bugs are tough ones frontier models missed at the start of 2026, not random trivia — yet Qwen reviewing a normal PR catches far more
- Running Qwen3.8-27B (8-bit) three times beats a single run of Opus 4.8 (max) in detection coverage, showing the payoff of repeated sampling for bug hunting
- Costs across the leaderboard span roughly 200x (log scale); runtime under 7x
The board is blind-graded with one prompt per repo; data and caveats are public on GitHub, and only runs the author executed are listed.
Related event: Bug Hunt Bench: local Qwen3.8-27B nears Claude Opus in bug fixing(3 posts)→
More from Models
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
- Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02