Epoch AI audits 15 AI benchmarks: 4 Verified, 9 Flawed; PostTrainBench passes and preps v1.2
maksym_andr · x · 2026-09-18
Epoch AI launched Benchmark Reviews, a new initiative auditing AI benchmarks, starting with 15: 4 Verified, 9 Flawed, and 2 with insufficient information. PostTrainBench v1.1 was designated Verified, and its team published a point-by-point response:
- On narrow overfitting to a single eval: this is intended — each agent gets one H100 for ten hours of post-training, so teaching broad transferable skills is unrealistic; the question is whether the agent improves the model on one target benchmark
- On middling averages and high variance: flagged runs are manually verified to keep the judge calibrated
- The team agrees with some findings and will release v1.2 addressing the flagged issues plus new ones found internally
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Models
- Musk cites unsustainable Grok Bot costs as Hermes offers one-command agent migration — Teknium · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- Same leak, reshared: ChatGPT's five family plan prices spotted again — heyshrutimishra · 2026-11-04
- KOL visits StepFun's Shanghai office, hints a powerful new model is coming — teortaxesTex · 2026-11-04
- Fable 5.5 delayed as Anthropic reportedly bets OpenAI's Astra 6.2 won't beat it — haider1 · 2026-11-04
- ByteDance open-sources Axion 1.0-35B agent foundation model with full reproducible recipe — pmttyji · 2026-11-04