Same model, new harness: success rate jumps from 68% to 88%
Arindam_1729 · x · 2026-11-05
Marmelab's State of Harness Engineering report: same model, tasks, and tools, but swapping the harness moved success rate from 68% to 88%. The team audited 246 agent repos and 57 publications, finding 60% of harnesses have no tests and no evals — just unverified AGENTS.md markdown.
Core thesis: a coding agent is model plus harness — what it sees, what it can touch, what checks its work, what it remembers, who decides it's done. The model is rented; the harness is owned. Teams chase 5% gains by swapping models while a 20-point gap sits in an untested rules file.
Four things to do before touching the model
- Write down 'done' criteria — if the agent can't check it, it guesses
- Turn your most important rule into a script or hook, not a sentence
- Keep 10 real tasks as a fixed eval, rerun after every harness change
- Delete rules the agent never uses — stale context is still context
The next wave of great dev tools will be better harnesses, not smarter models.
More from coding & agent
- Model selection is a routing problem: match model size to task ambiguity, says Together — togethercompute · 2026-11-05
- TinyFish becomes Fireworks' web search partner, free live web for open models — pstAsiatech · 2026-11-05
- Perplexity's Computer rebuilds SpaceX 3D engine visualizer from a single YouTube video — AravSrinivas · 2026-11-05
- a16z backs Phaze, Parsec founders' agent-ready remote desktop, in $8.75M seed — a16z · 2026-11-05
- Gemini API's computer use tool is now generally available — patloeber · 2026-11-05
- Hybrid Reasoning Agents paper: latent+text hybrid reasoning with big token savings — alex_verem · 2026-11-05