Full data table: all 89 runs of DeepSeek V4.1 Flash across 9 harnesses
PawelHuryn · x · 2026-10-20
Pawel Huryn published the per-run data behind his DeepSeek V4.1 Flash 1017 Bug Hunt benchmark: same model, 9 harnesses, 3 effort levels, each cell showing bugs fixed out of 105.
- Max effort: Claude Code 39.1, DeepSeek's harness 42.7, Pi 42.2, OpenCode 37.6
- High effort: Pi 40.4, Grok Build CLI 40.4, DeepSeek 39.2, OpenCode 2.0 38.6, among others
- Low effort: DeepSeek 34.8, Claude Code 34.4, Pi 33.8, OpenCode 32.8
- The table backs up his earlier conclusions and shows large cross-harness gaps plus significant run-to-run variance
Related event: Bug Hunt Bench: Agent harness choice swings DeepSeek V4.1 scores by points(10 posts)→
More from coding & agent
- AI That Works #79: Building Evals on Production Data — vikvang1 · 2026-10-21
- StopLoss: A Cheap External Monitor Predicts Coding-Agent Failures Early, at 10-100x Lower Cost Than LLM Judges — Jiarong Liang · 2026-10-21
- Meta's HACE jointly evolves specialized foundation models and LLM agents across three domains — meta · 2026-10-21
- Supabase launches Agent Prompts: agents that monitor your projects and alert on issues — dshukertjr · 2026-10-21
- Giving Pydantic AI agents cross-session memory with embeddings: a 4-step recipe — CountyTop5970 · 2026-10-21
- Pinokio goes multiplayer: local AI app launcher adds social features — cocktailpeanut · 2026-10-21