Delip Rao: private-set AI leaderboards give maintainers power to freeze out rivals
deliprao · x · 2026-10-14
Delip Rao flags a new fairness problem in public AI leaderboards. Since top rankings now attract attention and funding, benchmaxxing has become rampant, pushing leaderboards to adopt private or hybrid evaluation sets — which shifts power to the maintainers.
Writing from personal experience, he warns that a superior system can be indefinitely delayed in evaluation if maintainers have incentives to protect current top-k companies, meaning you may never get a fair shot regardless of quality. He declines to name the leaderboard involved.
More from Models
- Sonnet 5.5 landing on the free tier is more significant than people realize — adonis_singh · 2026-10-14
- Copilot keeps hitting company guardrails on tasks its chat mode handles fine — annetgriffin · 2026-10-14
- Gemini 4 Argon reportedly beats human baseline on all four stealth capability evals — scaling01 · 2026-10-14
- User praises Gemini thought summaries but wants full raw traces — QuixiAI · 2026-10-14
- Gemini 4 Argon lands in OpenCode as team's second most-used model — adonis_singh · 2026-10-14
- Claude Opus 5.5 vs Gemini Argon: Same 3D Island Prompt, Strikingly Different Results — adonis_singh · 2026-10-14