Raschka: benchmarking new LLMs is like using IQ tests for hiring
rasbt · x · 2026-11-18
Sebastian Raschka (rasbt) argues that comparing new LLM releases via benchmarks has become a bit like using IQ tests for hiring decisions. Benchmarks remain useful baselines, but beyond that he doubts how indicative they are of individual tasks and needs — implicitly including "that release yesterday."
The take echoes growing community skepticism that leaderboard scores diverge from real-world usage.
More from Models
- Open-source Karpo-Decision-31B tops 300-model Decision Index with zero-token, one-forward-pass answers — rohanpaul_ai · 2026-11-18
- GPT-6.2 backlash is real user experience, not astroturf: dev says Astra has quirks but Sol is a solid daily driver — pvncher · 2026-11-18
- LLM protein-folding test on fresh PDB entry lands near-random structures — MoAlQuraishi · 2026-11-18
- Every's founder pushes back: we re-tested the model to the wire, OpenAI gave us no heads-up — kieranklaassen · 2026-11-18
- HF trending: Thinking Machines' Inkling-2-Small, MiniMax H3-Image and Qwen3.8-27B surge together — victormustar · 2026-11-18
- US watchdog: four top AI chatbots helped plan terror attacks despite refusals — Dense-Following-9964 · 2026-11-18