Benchmarks are like perfume: why public leaderboards demand skepticism and private evals
intellectronica · x · 2026-11-09
intellectronica riffs on a famous quote: benchmarks are like perfume — you can smell them, but don't drink them. Public leaderboards are interesting to browse but contain limited (and sometimes misleading) signal; for real work, test on your own tasks.
The post quotes Hamel Husain clarifying a widely misread benchmark study: treating it as yet another meta-benchmark for picking models (open vs closed, model families) misses the point. The study's intent is to instill healthy skepticism of all public benchmarks — the only way to know what's "better" for your use case is running your own private evals.
More from Models
- SimpleHC cuts Hyper-Connections training overhead from 59–100% to just 2.4% — SonglinYang4 · 2026-11-09
- Unverified claim: Opus 5.5 has only 10B active params; dev hands projects to Qwen-4-Flash — teortaxesTex · 2026-11-09
- Running 8-25 Opus 5.5 threads all day used under 20% of a $200 Claude sub — cyrus_zei · 2026-11-09
- Grok Bot quietly throttled usage, then reset quotas after user backlash — xiaohu · 2026-11-09
- Two padding tokens doubled a hosted DeepSeek V4.1-Flash success rate, pointing to phase sensitivity — No_Pea_6625 · 2026-11-09
- Is Opus 5.5 nerfed? Users report thinking patterns changed on November 3 — rzk · 2026-11-09