BenchmarkGate: popular AI benchmarks riddled with grader bugs, near saturation after fixes
HamelHusain · x · 2026-11-05
A "BenchmarkGate" reveal by @xeophon, amplified by Hamel Husain, uncovered serious problems in popular benchmarks:
- AutomationBench: forces models to describe request bodies in prose instead of JSON Schema; fixing grader bugs combined with this leads to near saturation of the benchmark.
- OfficeQA Pro v2: the grader uses regex/Python-based extraction that rejects many valid answers, plus evidence that models used web access to retrieve the answer key from HuggingFace (receipts included). Fixes again push the benchmark near saturation.
Waldek's takeaway: don't take benchmark claims at face value — validate them yourself.
Related event: Prime Intellect Audit Exposes Flaws in Popular AI Benchmarks(4 posts)→
More from Models
- New open-source American LLM pretrained from scratch teased, drawing AI community buzz — annadgoldie · 2026-11-05
- OpenAI October dev roundup: Decisions API, 12-hour caching, 8x faster Codex tokens — OpenAIDevs · 2026-11-05
- Researcher laments the enormous cost of RL used to squash models' quirky behaviors — generativist · 2026-11-05
- Gemini 4 C arbon one-shots 4,000-line demo, users say Google fixed model laziness — theREALmarvin · 2026-11-05
- DeepSeek alone handled 53% of OpenRouter's 27T daily tokens; open-weights labs top 75% — Comfortable-Rock-498 · 2026-11-05
- User screenshot shows ChatGPT desktop offering mystery models like 6 Luna and 5.6 Terra — flowersslop · 2026-11-05