Meta eval lead's BenchBench chart shows self-reported eval counts across labs
ziqiao_ma · x · 2026-11-08
Responding to criticism that labs outsource evals, Meta's eval team member ziqiaoma shows her BenchBench chart: by self-reported model-card counts alone, her team reports more evals (85) than most compared labs (Claude Haiku 5.5: 7; DeepSeek V4.1 and GPT-6 Luna: 11 each). Third-party benchmarks complement internal work—private test sets guard against gaming, so labs use them to check held-out generalization, while maintaining internal implementations for public benchmarks.
More from Models
- Vercel lists DeepSeek V4.1 Flash 1017 at 1,000+ tok/s, but provider speed numbers don't add up — gajesh · 2026-11-08
- Researchers spar over why JEPA is hyped as 'the big thing in vision' among non-experts — ducha_aiki · 2026-11-08
- Tencent's Hy4 flagship lands on Chutes with vision, terminal control, $0.525/M input pricing — markjeffrey · 2026-11-08
- Kimi K3.1 Pro tops BabyVision image benchmark at 95.4%, beating Inkling 2 Small — NielsRogge · 2026-11-08
- World model licensing is a mess: open weights, restrictive decoders block commercial use — prateekj · 2026-11-08
- Gemini Won't Accept PPTX Files, but NotebookLM Does — Users puzzled — Pearl_Jam_ · 2026-11-08