Same Model, Different Harness: Benchmark Scores Swing by 35 Points, Chamath Says Leaderboards Are Meaningless

Chamath argues model leaderboards have lost their value, citing 8090 Labs research showing the same model's coding scores can swing from 24% to 59% depending on the harness used. Since real agents combine models with harnesses, he says benchmarks testing models alone no longer reflect actual capability.

2026-10-13 ~ 2026-10-13 · 2 related posts

• Swapping only the harness moves a model 35 points: 8090 Labs tests 7 models across 5 coding agents — pstAsiatech · 2026-10-13
• Same model scores 24% vs 59% across harnesses, Chamath declares leaderboards worthless — eliano · 2026-10-13