Same model, new harness: success rate jumps from 68% to 88%
Arindam_1729 · x · 2026-11-05
Marmelab's State of Harness Engineering report ran one model through 8 different harnesses on the same 25 coding tasks: success rate went from 68% to 88%. The team then audited 246 agent repos and catalogued 57 papers. Key finding: 60% of harnesses had no tests and no evals — just unverified markdown instructions in an AGENTS.md. Takeaway: harness engineering is the underrated lever; fix testing and evals before blaming the model.
More from coding & agent
- Axiom ships 16M-line AI formalization of the Stacks Project, countering 'AI code is slop' — Scobleizer · 2026-11-05
- Oumi launches FAiE, an agent that builds and trains your specialized AI agents in 15 minutes — JaynitMakwana · 2026-11-05
- Five ways an agent run can end: success, giving up, budget, human handoff, failure — ClickOk5811 · 2026-11-05
- Solo dev builds Suno + three.js music video pipeline with Claude Code in hours — Glass-Garden-5888 · 2026-11-05
- An obscure git command yields a 12% speedup, tested on the Linux kernel repo — DanielLockyer · 2026-11-05
- Visual Token Routing cuts VLM agent visual tokens up to 5.8x at same accuracy — wellecks · 2026-11-05