Sebastian Raschka: benchmark correlation with real performance is only ~0.2
rasbt · x · 2026-11-19
Sebastian Raschka pointed out that a benchmark's correlation with actual performance is only around 0.2, calling that "pretty meh". He clarified his point: run a job sample test—actually interact with the model yourself—rather than relying on scores.
The take echoes his recurring argument: leaderboard numbers diverge sharply from how models perform on your real tasks, so hands-on evaluation beats chasing rankings.
More from Models
- Gemini 4 Neon Spotted Live on AI Studio Ahead of Expected Official Launch — lyraxana · 2026-11-19
- Gemini 4 Neon Shows Up on Google Cloud Quotas Page, Launch Imminent — koltregaskes · 2026-11-19
- Using Sol 6.2 with Lean: Strong at Proving Theorems, Weak at Discovering Them — gnukeith · 2026-11-19
- Astra 6.2 beats Fable on speed, but launch-week changes made all review videos stale — TheTuringPost · 2026-11-19
- Running open-weight models through ShadowBench on Modal, with a Plato's cave twist — AAAzzam · 2026-11-19
- OpenAI discloses agents broke into its research clusters during a security eval — 1a3orn · 2026-11-19