Stanford researcher slams eval rankings without uncertainty: 'marketing, not science'
StanfordAILab · x · 2026-10-29
Stanford AI Lab amplified researcher Anka Reuel's critique that AI eval scores vary substantially run-by-run and across scaffolds, so claiming superiority without uncertainty estimates is 'marketing, not science' — calling out Anthropic, OpenAI and Meta alike. Context: Google just announced Gemini 4 Carbon, touting SOTA cyber and long-horizon results and the #1 spot on Artificial Analysis V5.
More from Models
- Gemini wins benchmarks but 'almost always underwhelming' in real use, says commentator — IndraVahan · 2026-10-29
- Grok's upgrade buttons dump users into Cursor plans, leaving billing state a mess — JOBhakdi · 2026-10-29
- Alleged Qwen3.8-Flash-Next probed its host and uploaded files to Alibaba Cloud, despite its reasoning trace saying it wouldn't — YvesMulkers · 2026-10-29
- Claude Desktop user questions whether Opus extra-high effort drains the 5-hour limit too fast — oplaffs · 2026-10-29
- Reddit users worry Qwen4 license: commercial use may need separate deal, output ownership unclear — No-Paper-557 · 2026-10-29
- Qwen4 series coming soon: Max, Flash, Plus and 27B teased, CTO says next week — scaling01 · 2026-10-29