OpenAI criticized for claiming 100 open math problems solved without disclosing the total attempted
burny_tech · x · 2026-09-22
marcusgnt publicly called out OpenAI's eval reporting: the company claims its model solved 100 open math problems, but never disclosed how many were attempted. He argues reporting the success rate is basic scientific practice and essential context for judging model ability, urging OpenAI to publish that simple number for the math community's sake. The main post is a repost endorsing this critique.
More from Models
- Robot build test: GPT-6 Astra stabilizes two-board structure, Claude Fable 5.2 falls short — qinzytech · 2026-11-12
- Zvi dissects Anthropic's latest system card, disputing the autonomy threshold call — Don't Worry About the Vase (Zvi) · 2026-11-12
- Unsloth ships Qwen3.8-27B Dynamic V3.1 GGUFs with 5-10% accuracy gains — danielhanchen · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12