Same model, three harnesses: DeepSeek harness scores 42.5 vs Claude Code 39.1 on Bug Hunt Bench
PawelHuryn · x · 2026-10-19
Pawel Huryn published full Bug Hunt Bench results for DeepSeek V4.1 Flash 1017 across harnesses: fixing 105 planted bugs, the DeepSeek harness averaged 42.5 (7 runs), Pi harness 42.2 (5 runs), and Claude Code 39.1 (7 runs) — Pi matching top scores at lower cost. Notably, Claude Code at high effort (36.4) underperformed max effort (39.1). Diffs were blind-judged by Grok; three runs that read answers from outside their folder were voided. High effort + Pi testing is next.
Related event: Bug Hunt Bench: Harness Choice Swings Model Repair Scores(4 posts)→
More from coding & agent
- Musk amplifies Grok Bot that phones restaurants and books tables for you — elonmusk · 2026-10-19
- Build a calibrated decision model on any LLM without retraining, Colab inside — kalyan_kpl · 2026-10-19
- Melony adds a free chat UI to share Claude Managed Agents with your team — DarasStayHome · 2026-10-19
- How 90% of software engineering looks in 2026: engineers just pass PRs between the same AI — Yuchenj_UW · 2026-10-19
- PM drowning in 10 routing pings a day builds a 'Company OS' in Slack that answers with sources — aakashgupta · 2026-10-19
- Codex now suggests follow-up prompts — the machine tells you what to ask next — burkov · 2026-10-19