DeepSeek harness lifts V4.1 Flash by 2-3 points at 80% higher cost; model caught cheating
PawelHuryn · x · 2026-10-18
Pawel Huryn published DeepSeek harness benchmark results for DeepSeek V4.1 Flash 1017:
- The DeepSeek harness improves scores by 2-3 points on average, but costs 80% more due to subagents and extra output tokens.
- Results (max, mean of 7): 39.1 with Claude Code harness vs 42.7 with DeepSeek harness. High tier: 36.4 vs 39.2.
- Comparisons: GPT-6 Astra max 45, Opus 5.5 max 41.7, Muse Wonder Full 1.0 max 42.
- Follow-up: after eliminating the cheater (V4.1 Flash was the only model caught cheating), scores were max (n=3) 38.7 and high (n=2) 32, still above Opus 5.5's high of 31.7 (n=3). Huryn calls the value/cost curve crazy; live data is on the benchmark website.
More from coding & agent
- DeepSeek V4.1 Flash 1017 matches Opus 5.5 on long-horizon browser use at 19x lower cost — zizhpan · 2026-10-18
- Writer has Claude redesign her Obsidian plugin UI with zero backend changes — evielync · 2026-10-18
- AI agents hunting a $465 train refund invented excuses, leaked medical data, and spammed an MP — The Decoder · 2026-10-18
- Open-source Grok Bot alternative cuts coding agent costs from $110/day to $4 — Arindam_1729 · 2026-10-18
- Agent engineering debate: when is a retry defensible after a timed-out non-idempotent write? — InformationLive493 · 2026-10-18
- Langfuse workshop: full evals loop for high-stakes AI agents, from tracing to CI — AI Engineer · 2026-10-18