Critic slams eval report for naming 36 benchmarks but never listing them, omitting rivals
gleech · x · 2026-10-12
Researcher gleech criticizes a model eval report's 'chart crimes': the biggest issue is that the 36 benchmarks used are never specified, making results unverifiable. He also notes the main chart omits Jev, and Sol's accuracy and Luna (including Luna Decisions) are excluded from comparisons, distorting the cross-model picture.
Related event: Model Evaluation Report Criticized for "Chart Crimes"(2 posts)→
More from Models
- ThursdAI: OpenAI drops 722 math papers, Haiku 5.5 hits 10 cents, Meta's new agent protocol — altryne · 2026-10-12
- Liquid AI's open d1-3B makes decisions in 8 milliseconds — classifier or new primitive? — altryne · 2026-10-12
- Mercury Decide nails 669 clinical decisions for 1 cent, ties #2 at 94.2% cheapest — MaziyarPanahi · 2026-10-12
- EmbeddingGemma 2 lands on Hugging Face, giving science projects a free accuracy boost — HildeKuehne · 2026-10-12
- Meta employees hint at a major new model launch, rumored as early as this week — kimmonismus · 2026-10-12
- LEAN proofs can be 'correct but alien' or exploit kernel bugs, researcher warns — gklambauer · 2026-10-12