89 benchmark runs across 9 harnesses: Pi and Grok Build CLI top DeepSeek V4.1 Flash
PawelHuryn · x · 2026-10-20
Pawel Huryn ran the DeepSeek V4.1 Flash 1017 Bug Hunt benchmark 89 times across 9 agent harnesses, 5 attempts each at high effort with a 3-hour per-repo time limit.
- High-effort means: Pi 40.4, Grok Build CLI 40.4, DeepSeek's own harness 39.2, OpenCode 2.0 38.6, OpenCode 36.6, Claude Code 36.4, Codex 33.2, Hermes Agent 32.0, Kilo Code 31.8
- Codex repeatedly exceeded the time limit and was scored on partial completion
- OpenCode was the weakest of the four main harnesses on this model; Pi offers the best value per cost
- Author's takeaway: Pi, Grok Build CLI and OpenCode 2.0 look like the strongest universal harnesses, though results may not generalize
Related event: Bug Hunt Bench: Agent harness choice swings DeepSeek V4.1 scores by points(10 posts)→
More from coding & agent
- Neo4j GraphAcademy launches AI proxy so workshop learners skip OpenAI billing setup — JeremyCMorgan · 2026-10-21
- Three weekend Claude Code projects: personal benchmarks, mini-SaaS, AI asset library — EXM7777 · 2026-10-21
- vLLM gains sleep mode on Apple Silicon: idle model servers now return KV cache memory — abhijithneil · 2026-10-21
- Open-source Diagram Design turns prompts and codebases into editorial-grade technical diagrams — bibryam · 2026-10-21
- Agent web-data startup raises $9M led by General Catalyst, 800+ customers — aftahi_ai · 2026-10-21
- AWS shows how to build conversational supply chain agents with Bedrock AgentCore and MCP — AWS ML Blog · 2026-10-21