Paper reveals Agent benchmark unreliability: Harness variance 7.8x model variance
omarsar0 · x · 2026-08-26
A new paper investigates why agent leaderboard comparisons are hard to trust, focusing on how much of a benchmark score is attributed to the "harness"—the layer between the model and the task.
The harness builds context, mediates tool calls, validates outputs, and decides when to stop. In a controlled experiment on 100 tasks from SWE-bench Verified, researchers tested three frontier models against three harness configurations while holding other variables constant.
Results showed that swapping the harness caused GLM-5.1's score to swing by 13.0 points, whereas swapping the model within a fixed harness caused changes of only 3.0, 2.5, and 5.0 points. Harness-induced variance was found to be 7.8x larger than model-induced variance, with 6 out of 9 model-pair rankings flipping depending on the harness used.
Related event: Paper finds agent benchmarks unreliable as harness matters more than models(2 posts)→
More from coding & agent
- Musk amplifies dev's verdict: Grok Bot easier to use than Codex and Claude — elonmusk · 2026-10-15
- Blob0 launches: a macOS file-manager-style desktop client for S3 and Cloudflare R2 — yihui_indie · 2026-10-15
- The 6-Layer AI Engineer Skill Stack for 2026: Vectors to Multi-Agent Ops — mdancho84 · 2026-10-15
- Building a Mechanical Engineering Agent Workflow in ChatGPT: 30-Node Canvas Shows Promise — burhop · 2026-10-15
- Open-source Harness Engineering course hits 19k stars, shows $9 vs $200 gap between failing and working agent setups — thisguyknowsai · 2026-10-15
- A 6-person team shipped healthcare prior-auth AI agents in 3 months — here's the playbook — alex_verem · 2026-10-15