Bug Hunt Bench Ranks Frontier Coding Models on 105 Real Bugs

Developer Pawel Huryn (author of The Product Compass) runs Bug Hunt Bench: 105 real bugs deliberately injected into two production codebases, with frontier models (GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM) asked to fix them in their agentic setups, blindly scored, one prompt per repo, results continuously updated on his GitHub leaderboard. This round's highlight is DeepSeek-V4.1-Flash's cost-effectiveness; Huryn is also retesting Kimi K3 and GLM at their highest reasoning tiers.

Confirmed

• GPT-6 Astra supplementary scores: low tier 27/105, medium tier 34/105, published Sep 4-5 and viewable via leaderboard filters.
• DeepSeek-V4.1-Flash (high tier): 24/105 fixed at max effort, total cost $1.80, 42.6 minutes.
• Huryn's retest shows V4.1 Flash clearly beats V4 Pro: DeepSeek V4 Pro (max) fixed only 16/105 at $1.89 and 37 minutes.
• Per secondhand accounts of the same bench: Claude Opus 5 (max) fixed 27 at $51.33; Grok 4.6 (max) and other flagships are also compared; Flash's $1.80 is 1/28 (3.5%) of Opus's cost.
• Kimi K3 and GLM retests at highest reasoning tiers are in progress, results not yet posted.

Unconfirmed

• The Opus/Grok comparison figures come from reposted summaries; the GitHub leaderboard is the authoritative source.

Why it matters

• Real repos, injected bugs, blind scoring, and single-prompt setup make this closer to real-world debugging than synthetic benchmarks.
• Flash fixing 24 bugs for $1.80, beating its own pricier V4 Pro, offers valuable selection data for budget-sensitive teams.
• Continuous retesting may turn the leaderboard into a living reference for frontier coding models' bug-fixing ability.

2026-09-10 ~ 2026-09-11 · 13 related posts

Full story(18 episodes)→

• Episode 1: DeepSeek V4.1 Flash Opens Limited Internal Beta with New Architecture and Native Multimodality(2026-09-08, 18 posts)
• Episode 2: DeepSeek V4.1 Flash Tested: Blazing 350 Tokens/s but Still Experimental(2026-09-08, 2 posts)
• Episode 3: DeepSeek cuts V4-Flash API prices with new peak/off-peak billing from Sept 10(2026-09-08, 6 posts)
• Episode 4: DeepSeek V4.1 Flash Nears Frontier Models at a Fraction of the Cost, Independent Tests Show(2026-09-09, 15 posts)
• Episode 5: DeepSeek Releases Open-Source V4.1 Flash, Beating Its Own Flagship at a Fraction of the Cost(2026-09-09, 57 posts)
• Episode 6: DeepSeek V4.1 Flash Leaks: 552B Asymmetric MoE Claimed to Rival GPT-5.6 Sol(2026-09-10, 21 posts)
• Episode 7: DeepSeek V4.1 Flash Benchmark Results Leak, Sparking Community Discussion(2026-09-10, 2 posts)
• Episode 8: DeepSeek Shrinks Per-Token KV Cache 54x in Nine Months to Sub-KB Levels(2026-09-10, 6 posts)
• Episode 9: Leaked DeepSeek-V4.1-Flash Touted as New Open-Source King(2026-09-10, 3 posts)
• Episode 10: Bug Hunt Bench Ranks Frontier Coding Models on 105 Real Bugs(2026-09-10, 13 posts)
• Episode 11: DeepSeek V4.1 Flash Confirmed to Center on YOCO Architecture(2026-09-10, 7 posts)
• Episode 12: Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship(2026-09-11, 44 posts)
• Episode 13: DeepSeek V4.1 Flash tops Vals open-source index at $0.30 per run(2026-09-11, 2 posts)
• Episode 14: DeepSeek launches V4.1-Flash: 552B backbone, 1M-token context(2026-09-11, 5 posts)
• Episode 15: DeepSeek V4.1 Flash scores 40 in AA index despite rising hallucinations(2026-09-11, 2 posts)
• Episode 16: Inside KV-Cache Sharing: How DeepSeek CED and GLM 5.2 Differ(2026-09-11, 4 posts)
• Episode 17: DeepSeek 4.1 Flash Shows Big Gains in Kernel Engineering and Cost(2026-09-11, 2 posts)
• Episode 18: SGLang delivers Day 0 support for DeepSeek V4.1 Flash, hitting 873 tok/s(2026-09-12, 2 posts)

• [source] Bug Hunt Bench grades frontier models on 105 real bugs; DeepSeek-V4.1-Flash lands 24/105 for $1.80 — PawelHuryn · 2026-09-10
• Bug Hunt Bench adds GPT-6 Astra scores: 27/105 low, 34/105 medium — PawelHuryn · 2026-09-10
• Same Bug Benchmark: GPT-6 Astra Medium Fixes 34/105, Low Scores 27 — PawelHuryn · 2026-09-10
• DeepSeek V4.1 Flash fixes 24 of 105 hidden bugs at $1.80, near Opus-level at 1/28th the cost — AlexSed90 · 2026-09-10
• [source] DeepSeek V4.1 Flash scores 24/105 for $1.80 in community eval test — PawelHuryn · 2026-09-10
• Bug Hunt Bench ranks frontier coding models on 105 real planted bugs, with 200x cost spread — PawelHuryn · 2026-09-10
• DeepSeek V4.1 Flash fixes 24/105 planted bugs for $1.08 in retest — sunncynn · 2026-09-10
• DeepSeek V4.1 Flash beats V4 Pro in retest: 24/105 vs 16/105 at lower cost — PawelHuryn · 2026-09-10
• Bug Hunt Bench retests Kimi K3 and GLM on planted real-world bugs — PawelHuryn · 2026-09-11
• DeepSeek V4.1 Flash fixes 24 of 105 planted bugs for $1.80, near Opus 5's $51.33 run — iPaulis · 2026-09-11
• [source] Bug Hunt Bench ranks frontier coding models on 105 planted real-repo bugs — PawelHuryn · 2026-09-11

2 near-duplicate retellings: PawelHuryn · ChartsJournalX