28,849 trials and $4,746 later: a leaderboard ranks coding-agent harnesses on open models
HankYeomans · x · 2026-11-06
- Harness Leaderboard released: joelniklaus ran 28,849 trials ($4,746 of inference) to answer which coding-agent harness to use with open models, comparing nine harnesses.
- Controlled setup: only the harness changes per comparison — model, provider and reasoning effort stay fixed, each task gets three attempts. Models tested: GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE.
- Key result: averaged over the three models, bash-only mini-swe-agent (@KLieret) solves 51.4% of tasks vs 27.3% for openclaw; four other harnesses are statistically tied with mini-swe-agent, including Pi (@pidot). pidotdev notes cost-per-task is a critical metric for their team.
Related event: 28,849 Tests Rank 9 Coding Agent Frameworks, 24-Point Gap Found(4 posts)→
More from coding & agent
- Musecases now catalogs 600+ real-world Meta Muse AI agent use cases — thisiskp_ · 2026-11-06
- Claude Code can now edit videos — first preview in under 20 minutes — PrajwalTomar_ · 2026-11-06
- 0.1B SPLADE sparse retrieval beats Qwen3-0.6B dense embeddings in local RAG tests — jeffrey-0711 · 2026-11-06
- Siraj Raval: better plumbing beats a bigger model in his code-free VR museum build — sirajraval · 2026-11-06
- Documentation Is Part of Done: Put It in Your Definition of Done — dSebastien · 2026-11-06
- BAAI's AREX-2: a 27B open-weights model that runs its own try-check-retry agent loop — Sam Witteveen · 2026-11-06