105 Hidden Bugs Benchmark: DeepSeek V4.1 Flash Scores 24 at $1.80 vs Opus 5's $51
sanderjson · x · 2026-09-10
PawelHuryn ran a real-world benchmark across 2 repos with 105 hidden bugs, asking models to find and fix them:
- Opus 5 (max): 27 fixed, costing $51.33
- Grok 4.6 (max): 27, $16.96
- DeepSeek V4.1 Flash (max): 24, only $1.80
- GPT-5.6 Luna (xhigh): 23, $2.50
- Opus 5 (high): 21, $38+
Takeaway: DeepSeek V4.1 Flash is remarkably strong for everyday coding tasks — nearly matching frontier models at roughly 1/28th the cost of Opus 5.
More from coding & agent
- DeepMind paper: LLMs can't sample randomly on their own, only convert given seeds — rohanpaul_ai · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- BELL: Open-Source Stack Combines Browserbase, Eve, iMessage and Stripe Link for Personal Agents — jeff_weinstein · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Two engineers rebuilt a FedEx supplier's billing core in 3.5 months vs an estimated 7-8 — alex_verem · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27