LLM leaderboards are now often measuring the harness too, Gary Marcus warns
GaryMarcus · x · 2026-07-22
Gary Marcus amplifies a point that matters for model evaluation: teams have quietly shifted from benchmarking LLMs alone to benchmarking LLM + harness, while keeping the same leaderboard labels.
The core complaint is methodological: if the system under test changes but the benchmark name does not, comparisons become misleading. Marcus says this is important and people need to understand it to grasp what is happening in the field.
More from Research
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- Hugging Face wraps 6-month Training Agents series with open recordings and code — SergioPaniego · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26