Paper: Harness-Induced Variance Is 7.8x Model Variance in LLM Agent Benchmarks
rohanpaul_ai · x · 2026-09-01
A new paper, "Stop Comparing LLM Agents Without Disclosing the Harness," argues that for long-horizon agents the harness can matter more than the model, so benchmark scores shouldn't be compared without disclosing or controlling the harness.
In a controlled 3-model × 3-harness study on 100 SWE-bench Verified tasks:
- Average harness-induced variance was 7.80x model-induced variance.
- Holding the model fixed, moving from minimal to full harness changed pass@1 by 8.5–13.0 points; holding the harness fixed, switching models changed it by only 2.5–5.0 points.
- Rankings were unstable too: 6 of 9 model-pair/harness-pair comparisons reversed under another harness.
Paper: arxiv.org/abs/2605.23950
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Pi Durable went through 7 rewrites yet agent session files still lack a standard — ricklamers · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18