Coding-Agent Harness Leaderboard Adds Consistency Metric: Only a Third of Tasks Solved Every Time
OfirPress · x · 2026-11-08
Joel Niklaus's Harness Leaderboard now tracks how often each coding-agent harness solves the same task on all three attempts — a key promise for unattended work.
Averaged over three open models, mini-swe-agent leads with 51.4% per-attempt success, 68.6% success at least once, but only 33.0% on every attempt. Five harnesses cluster at 32.0–33.3% consistency, too close to call; openclaw manages just 15.3%.
On the weakest model, Flint, NousResearch's Hermes Agent solves 28.6% at least once but only 10.0% every time — showing consistency degrades sharply on weaker models.
More from coding & agent
- Dev builds free browser tycoon game Mall Bats with Claude writing nearly all the code — jacobjr23 · 2026-11-09
- Open-Source Mac Status Light App Monitors Multiple Claude Code Sessions — stateofteddy · 2026-11-09
- Researcher builds hardest-ever RCE chain largely AI-driven with GLM — banteg · 2026-11-09
- Dad and kids build a playable Roblox game with zero code using Claude, MCP and Tripo — minchoi · 2026-11-09
- You can still spot a vibecoded site in 2 seconds, even with Opus 5.5 — hewarsaber · 2026-11-09
- Terminal-RSI boosts 27B model's terminal-agent success from 20.7% to 50.6% via recursive self-improvement — Dr_Singularity · 2026-11-09