Harness Errors Cause 25% Score Swings, Exposing Flaws in Raw Benchmarks
Imaginary_Dinner2710 · reddit · 2026-07-31
The author points out that current LLM benchmarks are highly dependent on the test harness. On the ARC-AGI benchmark, an OpenAI model scored only 13% due to improper API calls, but jumped to 38% when corrected.
In real-world development, models are always used within specific agent frameworks (like Claude Code). Testing raw models without these harnesses is meaningless, fails to reflect actual capabilities, and misleads developers. The industry needs to move away from this unrepresentative evaluation method.
Related event: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(9 posts)→
More from Models
- Beam's Ten-Benchmark Breakdown: Doubles Peers on CritPt, Near-Bottom on Terminal-Bench — ArtificialAnlys · 2026-10-16
- Beam Scores 28 on Coding Agent Index, 40 Behind Leaders but Among Most Token-Frugal — ArtificialAnlys · 2026-10-16
- Beam Token Breakdown: Only 26% of Its 26k Output Tokens per Task Are Answers — ArtificialAnlys · 2026-10-16
- Reflection's First Model Beam Scores 30, Ranks #2 Among US Open Weights Models — ArtificialAnlys · 2026-10-16
- Why are sub-15B open models so rare now? Labs seem to have settled on 24-27B minimum — Hot_Example_4456 · 2026-10-16
- User spots model openly generating "ideal responses" to train other models — liminal_bardo · 2026-10-16