Pi harness nearly matches DeepSeek's own at 42.2 vs 42.5 on Bug Hunt Bench, at half the cost
PawelHuryn · x · 2026-10-19
Pawel Huryn reports new Bug Hunt Bench results: running DeepSeek V4.1 Flash 1017 (MAX effort) in the Pi harness scores 42.2 (mean of 5), nearly matching DeepSeek's own harness at 42.5 (mean of 7) — at $0.83 vs $1.84 per run. The Claude Code harness scores 39.1 with the same model. Takeaway: the harness itself materially affects benchmark scores, and Pi achieves comparable results at roughly half the cost. High effort + Pi testing is next.
Related event: Bug Hunt Bench: Harness Choice Swings Model Repair Scores(4 posts)→
More from coding & agent
- Musk amplifies Grok Bot that phones restaurants and books tables for you — elonmusk · 2026-10-19
- Build a calibrated decision model on any LLM without retraining, Colab inside — kalyan_kpl · 2026-10-19
- How 90% of software engineering looks in 2026: engineers just pass PRs between the same AI — Yuchenj_UW · 2026-10-19
- PM drowning in 10 routing pings a day builds a 'Company OS' in Slack that answers with sources — aakashgupta · 2026-10-19
- Codex now suggests follow-up prompts — the machine tells you what to ask next — burkov · 2026-10-19
- Same prompt, three designs: Claude vs ChatGPT vs Figma Make UI face-off — Tegadesigns · 2026-10-19