AIFlow slams small-n evals: "Four tasks, one run each, no error bars — a vibe check with a logo"

AIFlow_ML · x · 2026-11-20

AIFlowML criticized a common pattern in AI model evaluations: running just four tasks, once each, with no error bars. The snarky takeaway — such benchmarks are statistically meaningless, "just a vibe check with a logo" — underscores ongoing concerns about tiny sample sizes and lack of rigor in public LLM benchmark comparisons.

Original post →

More from Models

Models channel →