Same model scores 24% vs 59% across harnesses, Chamath declares leaderboards worthless
eliano · x · 2026-10-13
Chamath argues model benchmarks are largely worthless: the same model can score 24% or 59% on the same 113 coding tasks depending on the harness wrapping it. He had 8090 Labs evaluate 7 models across 5 harnesses on real bug fixes and features from open-source projects, measuring the model-plus-harness pair.
Surprising results include Claude Fable 5.1 scoring 41% inside Claude Code but 61% inside OpenAI's Codex, with a median of 28 tool calls. He says he increasingly ignores leaderboard results since ranking gains don't show up in real work.
The post also quotes Palantir CTO Shyam Sankar: benchmarks measure something real, but not the tasks enterprise customers actually need solved.
More from Models
- Muse vs Opus 5.5: same 3D prompt costs $4.11 vs $99.43, big quality gap — iamaliveix · 2026-10-13
- 179M-Parameter Open Retriever UIBK-Embedding-v1 Tops BEIR Under 1B Models — antoine_chaffin · 2026-10-13
- ProofBench leaderboard: Claude models hit 100%, 10th-place system named 'Wonder Full' — shuchaobi · 2026-10-13
- LiquidAI ships MLX versions of d1 models optimized for Apple Silicon local inference — JosephJacks_ · 2026-10-13
- Unverified: Zuck said to ship a new Muse version every month, each 'exponentially better' — haider1 · 2026-10-13
- Muse Wonder defended as a real code model, backed by 10,565 commits in a year — ren_hongyu · 2026-10-13