Raschka: benchmarking new LLMs is like using IQ tests for hiring

rasbt · x · 2026-11-18

Sebastian Raschka (rasbt) argues that comparing new LLM releases via benchmarks has become a bit like using IQ tests for hiring decisions. Benchmarks remain useful baselines, but beyond that he doubts how indicative they are of individual tasks and needs — implicitly including "that release yesterday."

The take echoes growing community skepticism that leaderboard scores diverge from real-world usage.

Original post →

More from Models

Models channel →