One kitten dies every time you cherry-pick a single benchmark run
remilouf · x · 2026-10-24
remilouf calls out a common practice in model launches: presenting a single (best?) benchmark run to claim your model is better. The quip targets a real flaw in current eval culture — without variance across multiple runs, a lone result proves little.
More from Models
- dottxt teases Tool-Call Index: worst provider breaks tool calls 17x more than best — remilouf · 2026-10-24
- New Gyro quant format accused of being a closed-source EXL3 clone with no proven gains — nasone32 · 2026-10-24
- Haider dismisses 'OpenAI falling behind' narrative: strongest model still unreleased — haider1 · 2026-10-24
- Two frontier models disagreed sharply on iHeartMedia's fate until layered reasoning converged them — AI Engineer · 2026-10-24
- Why does one f16 token take 64KB on Qwen3 8B? A KV cache explainer question — Fancy-Snow7 · 2026-10-24
- Grok bot cleared for EU while OpenAI, Anthropic and Meta rivals await approval — kimmonismus · 2026-10-24