28,849 trials benchmark 9 coding-agent harnesses: 24-point gap with same model
victor_explore · x · 2026-11-06
Joel Niklaus ran 28,849 trials across 9 coding-agent harnesses with the same model, provider, and reasoning effort, revealing a 24-point spread: mini-swe-agent leads at 51.4% macro accuracy ($0.17/trial) while OpenClaw bottoms out at 27.3%. Pi dominates the cost frontier at $0.094/trial ($0.29/solve). Kimi Code (50.5%), Hermes Agent (50.3%), and Claude Code (49.2%) round out the top five. Total cost: $4,746; all trajectories public with confidence intervals — proof that harness choice matters as much as model choice.
Related event: Developer Benchmarks 9 Coding Agent Harnesses Across 28,849 Trials(3 posts)→
More from coding & agent
- Formal proofs of tens of millions of lines of code are now feasible, says Anthropic engineer — morqon · 2026-11-06
- Terminal-Bench's two broken tasks traced to Debian 11 archive repo migration — PawelHuryn · 2026-11-06
- Stable AI proposes DPDM: frozen LLM representations plus shared predictor boost agent decisions — stable-ai · 2026-11-06
- One prompt ran Codex for 17+ hours with 109 sub-agents and 2B tokens — andimarafioti · 2026-11-06
- Reddit devs compare agent memory approaches: summaries, MEMORY.md, wikis and vector DBs all fail differently — ombre-ai · 2026-11-06
- MCP vs API vs CLI: build both on one operations layer, prioritize MCP — hottown · 2026-11-06