Sebastian Raschka: benchmark correlation with real performance is only ~0.2

rasbt · x · 2026-11-19

Sebastian Raschka pointed out that a benchmark's correlation with actual performance is only around 0.2, calling that "pretty meh". He clarified his point: run a job sample test—actually interact with the model yourself—rather than relying on scores.

The take echoes his recurring argument: leaderboard numbers diverge sharply from how models perform on your real tasks, so hands-on evaluation beats chasing rankings.

Related event: Meta-Analyses Find Intelligence Predicts Job Performance at Only ~0.2 Correlation(2 posts)→

Original post →

More from Models

Models channel →