Benchmaxxing score flags models ranking higher on public benchmarks than held-out ones
airesearch12 · x · 2026-10-20
The author argues a good benchmark must not be gameable, and introduces a technique to detect benchmaxxed models by identifying:
- Jagged scores across benchmarks of the same topic
- Models ranking higher on famous public benchmarks than on held-out ones of the same topic
- Low scores on tests published after training completed
This is implemented as the Benchmaxxing score on BenchmarkHeaven. Example screenshots show one model at -5.0 (no benchmaxxing signal) versus another at +12.7 (clearly outperforming on famous boards, +53 percentile points on Vision), with the caveat that the signal is a screening flag, not proof of leakage or contamination.
More from Models
- PrismML's Ternary Bonsai 2 squeezes Qwen3.5 VL models to ternary weights, 9B at 2GB — lmoroney · 2026-10-20
- Cline: MiniMax M3.1 Flash matches Opus and GPT on coding at roughly 25x lower cost — MiniMax_AI · 2026-10-20
- Gemini 4 'Argon' spotted in Gemini App and AI Studio, unconfirmed — Efficient-Break-6333 · 2026-10-20
- Anthropic Safeguards blog: Claude self-assessment flags unsafe chats, 78% recall at 1% FPR — javirandor · 2026-10-20
- Unverified: Gemini 4 Argon reportedly catching up with Opus 5.5 — vedantmisra · 2026-10-20
- Researcher: OpenAI appears roughly two months ahead of Anthropic on capability trend — peterwildeford · 2026-10-20