Leaderboard Shifts After Removing SimpleQA
scaling01 · x · 2026-07-19
A screenshot shows a model leaderboard where removing SimpleQA alters the scores and rankings of multiple models. For instance, GPT-5.6 Sol retains the top spot, while models like Claude, Gemini, and Kimi see varying degrees of upward or downward movement. The poster uses this to call out individuals who are "fabricating numbers" to arbitrarily downgrade ECI in order to force a specific conclusion.
Related event: Debate Over SimpleQA's Impact on Model Rankings(3 posts)→
More from Models
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26