AIFlow slams small-n evals: "Four tasks, one run each, no error bars — a vibe check with a logo"
AIFlow_ML · x · 2026-11-20
AIFlowML criticized a common pattern in AI model evaluations: running just four tasks, once each, with no error bars. The snarky takeaway — such benchmarks are statistically meaningless, "just a vibe check with a logo" — underscores ongoing concerns about tiny sample sizes and lack of rigor in public LLM benchmark comparisons.
More from Models
- Creating good evals may be the most important post-training job in coming years — joecole · 2026-11-20
- Voice receptionist demo shows GPT-Live-1.1 booking appointments and escalating urgencies — amos_gyamfi · 2026-11-20
- Leaked: X Preparing Black Friday Discount Bundling Premium with Grok and Cursor — nima_owji · 2026-11-20
- Inkling Small 2 evals: 97% flags flakiness, never implies causality — keirp1 · 2026-11-20
- Daily AI brief: Anthropic +17% on Nasdaq debut, Genie 4 opens to developers — testingcatalog · 2026-11-20
- SenseNova-Loop 9B runs on 8GB VRAM at 22 t/s, uses half the tokens of Qwen3.8 — R_Duncan · 2026-11-20