Swapping only the harness moves a model 35 points: 8090 Labs tests 7 models across 5 coding agents
pstAsiatech · x · 2026-10-13
Chamath cited new research from 8090 Labs arguing that model-only leaderboards miss half the picture: every coding agent pairs a model with a harness — the program that writes prompts, hands it tools, edits files, and decides what to do when a step fails. Teams pick the two together.
The study ran 7 models (GPT-5.6 Sol/Terra/Luna, Claude Fable 5.1, Grok 4.6, GLM 5.3 Flash, DeepSeek V4 Flash) across 5 harnesses (Codex, Claude Code, OpenCode, Pi, DeepSeek harness) on 113 real coding tasks from Datacurve DeepSWE v1.1, each combination twice, graded by Datacurve's own tests.
Key finding: holding model and reasoning settings fixed, changing only the harness moved scores by up to 35 points — or barely at all for some models. Claude Fable 5.1 scored 41% in Claude Code but 61% in Codex. Picking a model/harness combo off a model-only leaderboard means making a critical decision on half the information.
More from coding & agent
- Intent Launches: Manage Swarms of Coding Agents From Your Phone — Wattenberger · 2026-10-13
- Give agents a next step: embed fix steps and doc links in API error responses — SucceededMind · 2026-10-13
- Temporal agent falls for prompt injection but scoped authority blocks the .env read before it runs — daniel_tenuo · 2026-10-13
- Simple config.toml tweak reportedly lifts 272k coding context cap to 872k — JayJeds · 2026-10-13
- EpochGo: local-first multi-agent AutoML on Ollama, $0 API costs and 100% offline — Grouchy-Excitement28 · 2026-10-13
- Ramp CTO to Discuss Internal Coding Agent Behind ~75% of Merged PRs — graceisford · 2026-10-13