Paper Reveals LLM Leaderboards Are Flawed: Test Harness Matters More Than Models
sven_ai · x · 2026-08-26
A new paper argues that for long-horizon tasks, the agent execution harness is often a stronger determinant of performance than the model itself. Controlled experiments (3 models x 3 harnesses x SWE-bench) show that harness-induced variance is 7.8x greater than model-induced variance and causes ranking reversals. The paper proposes a 'Binding Constraint Thesis' and suggests that leaderboard comparisons should disclose harness specifications.
Related event: Paper finds agent benchmarks unreliable as harness matters more than models(2 posts)→
More from Research
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Google's HARNESS² evolves agent harnesses without verifiers, up to +11.5 points on WorkBuddy-Bench — dair_ai · 2026-10-15
- Subbarao Kambhampati to dissect the 'mythos of LLM thinking tokens' at UMN — rao2z · 2026-10-15
- How 3D Gaussian Splatting gives robots spatial memory of what's out of view — TheTuringPost · 2026-10-15