28,849-trial Harness Leaderboard: bash-only mini-swe-agent tops coding harnesses
yb2698 · x · 2026-11-06
Joel Niklaus spent $4,746 on inference across 28,849 trials to answer which harness to run open models in, releasing the Harness Leaderboard.- Compares 9 coding-agent harnesses; each comparison swaps only the harness while model, provider, and reasoning effort stay fixed, 3 attempts per task- Models: GLM-5.3-Flash, DeepSeek-V4.1-Flash, Flint; benchmarks: Terminal-Bench 2.1 and DeepSWE- Averaged over the three models, bash-only mini-swe-agent solves 51.4% of tasks vs. openclaw's 27.3%; four others are statistically tied with mini-swe-agent- Pi hits 49.4% at $0.29 per solved task, 45% cheaper than mini-swe-agent's $0.53- Best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent on DeepSeek. Takeaway: harness choice meaningfully shifts open-model performance and cost-efficiency.
More from coding & agent
- Yukon syncs tables near-real-time at 1M events/sec, built so agents can query what changed — testingcatalog · 2026-11-06
- How one creator turned X posts into 40,000 email subscribers with a Go/No-Go AI agent — VibeMarketer_ · 2026-11-06
- Three open-source models combine into a fully local, real-time voice-to-avatar assistant — shinjiw_at_cmu · 2026-11-06
- Garry Tan: Capy's coordinating subagents boosted his PR output ~13x — garrytan · 2026-11-06
- GitHub Copilot's agent stack: cross-repo projects, parallel sessions, desktop app and automations — 0xkarasy · 2026-11-06
- Stripe opens its data to agents: free queries through Jan 2027, SQL shown with every answer — emilygsands · 2026-11-06