"Evals Don't Reflect Real Usage": Doctor Vows Off AA Index as Rankings Grow Absurd
DrDatta_AIIMS · x · 2026-09-05
Harveen Chadha argues that benchmark scores increasingly fail to represent real-world user experience. "The day Opus 5 scored more than Fable 5 on AA was the day I stopped believing in the AA index," he writes, noting that Muse Spark 1.3 now beating Astra shows things are getting worse.
DrDattaAIIMS joked along in reply ("so I should quit my job?" 🤡). Core point: the gap between AI leaderboards and actual usage keeps widening, undermining their credibility.
Related event: Users Report Real-World Results Contradict AA Index Rankings(5 posts)→
More from Fun
- DevRel evolves: Developer Relations → Builder Relations → Agent Relations — Saboo_Shubham_ · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- tszzl on Starship Troopers: the satire meme works because it's 'literally coolest thing ever' — tszzl · 2026-10-21
- DeepSeek Flash generates impressive pixel art using only code — victormustar · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21
- AI founders' cold pitches are getting absurd: 'folding the universe through a wormhole' — altryne · 2026-10-21