Benchmark criticized for cherry-picking known failures; same model scores 75% vs 56% across harnesses
tallmetommy · x · 2026-11-12
Replying to Gary Marcus, a developer argues a widely cited AI capability benchmark was built from tasks where users had to rephrase or roll back — i.e., assembled from known failures — so its 26.4% figure is a hard-case rate, not the field-wide failure rate.
A wilder finding: the same model scored 75% with one harness and 56% with another, showing how much the evaluation framework itself shapes results.
More from Models
- Which Claude model for which job: a 5-scenario real-work comparison — petergyang · 2026-11-12
- Zvi 周报:METR 加速报告引发 2027 智能爆炸预期大幅更新 — Don't Worry About the Vase (Zvi) · 2026-11-12
- Anthropic user shows off 4 free limit resets: one full reset plus three Fable resets — zsakib_ · 2026-11-12
- Raon-1 Nano 3.1B gets native ik_llama.cpp runtime with 17% faster MTP speculative decoding — Any_Bus9205 · 2026-11-12
- New complexity paper claims more reasoning tokens strictly expand what transformers can solve; critics push back — gerardsans · 2026-11-12
- LibreAGI: A New Project to Save Open-Weights OSS Models — satuke · 2026-11-12