Benchmaxxing may be dead as top AI leaderboard keeps 60% of questions secret
teortaxesTex · x · 2026-10-29
@ziwenxu argues benchmaxxing is dying: the biggest AI scoreboard now keeps 60% of its questions secret, so labs can't train on the test, and rankings should finally reflect real-world usefulness. @DanielFein7 pushes back: well-specified coding tasks were simply the easiest environments to build, so all models got that capability; the goalpost is now moving to poorly specified tasks and science/math, and the real question is what labs choose to prioritize.
More from Models
- Google officially announces Gemini 4 Carbon — aigclink · 2026-10-29
- Company tries to sign up for new Gemini, gives up after an hour of inscrutable enterprise setup — anshulkundaje · 2026-10-29
- User reports Anti-gravity bug: agent burned 60% of weekly allowance in an hour with no output — Familiar_Somewhere35 · 2026-10-29
- GPT-6.1 Terra vs Sol: enterprises want cheap-good-enough models, says analysis — haider1 · 2026-10-29
- Users complain Grok's @bot appears nerfed: "this one is dumb" — alexcovo_eth · 2026-10-29
- Steve Hsu: Chinese labs neglect science/math, focus on coding+agentic as DeepSeek shines on FrontierCode — teortaxesTex · 2026-10-29