Leaderboard Shifts After Removing SimpleQA
scaling01 · x · 2026-07-19
A screenshot shows a model leaderboard where removing SimpleQA alters the scores and rankings of multiple models. For instance, GPT-5.6 Sol retains the top spot, while models like Claude, Gemini, and Kimi see varying degrees of upward or downward movement. The poster uses this to call out individuals who are "fabricating numbers" to arbitrarily downgrade ECI in order to force a specific conclusion.
Related event: Debate Over SimpleQA's Impact on Model Rankings(3 posts)→
More from Models
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- giffmana: the env being used in training is part of the point — giffmana · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11