DeepSeek V4.1 Flash fixes 24/105 real bugs for $1.80 vs Opus 5's $51.33
PawelHuryn · x · 2026-09-10
Pawel Huryn tested frontier models on 105 hidden real bugs across 2 repos (find-and-fix), with results and costs:
- Opus 5 (max): 27 bugs, $51.33
- Grok 4.6 (max): 27 bugs, $16.96
- DeepSeek V4.1 Flash (max): 24 bugs, $1.80
- GPT-5.6 Luna (xhigh): 23 bugs, $2.50
- Opus 5 (high): 21 bugs, $38
Methodology FAQ: bugs are real, tough issues frontier models struggled with in early 2026; unplanted bugs don't count (OpenAI models report many real-but-irrelevant issues that would inflate scores); each model is judged by a different model family with confirmed calibration; answer keys aren't public, anonymized runs linked on GitHub. Luna (max) is called crazy good and cost-effective; Muse Spark 1.3 went untested due to failed payments in Europe and unreliable OpenRouter effort levels. Verdict: DeepSeek V4.1 Flash is a really strong everyday model at an extremely low cost.
More from Models
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27
- Model refuses wrong psychiatric advice in deliberately misleading test — FellMentKE · 2026-10-27
- Clinical System One v0.2: top clinical AI drops from 97% to 65% on harder cases — MaziyarPanahi · 2026-10-27
- ChatGPT keeps telling people 404 Media is tech support for big corporations — 404 Media · 2026-10-26