Hamel Husain: AI benchmark companies are in the same business as credit rating agencies
HamelHusain · x · 2026-11-02
Hamel Husain argues that Gartner vendor rankings, credit rating agencies for mortgage-backed securities, US News college rankings, and AI benchmark companies all run essentially the same business. Most people haven't caught onto this about AI benchmarks yet, he says, but they will—a pointed critique of leaderboard credibility in the model evaluation industry.
More from AGI Musings
- The widest gap now is between tech and non-tech workers, says AI researcher — emax · 2026-11-02
- Ethan Mollick runs a 'reverse CAPTCHA': AIs swarm to Oxyrhynchus Papyri replies — eldonredwards · 2026-11-02
- Ethan Mollick: GPT-4 could work solo for 4 minutes; today's models run for weeks — eldonredwards · 2026-11-02
- Robotics isn't 'solved' by benchmark jumps — it's solved when GDP grows 100x — eigenron · 2026-11-02
- DeepSeek researcher: 'Don't make any mistakes' is the underrated moat — Yuchenj_UW · 2026-11-02
- Mathematician: LLMs matter for massive proofs, not short insight papers — littmath · 2026-11-02