ML Commons launches 'benchmarks as a service' to kill benchmaxxing
RexDouglass · x · 2026-10-30
ML Commons has quietly rolled out "benchmarks as a service," designed to evolve benchmarks as they saturate and put an end to benchmaxxing. An accompanying chart shows saturation indices across safety benchmarks: SimpleSafetyTests is fully saturated at 1.00, OR-Bench at 0.93, SORRY-Bench 0.90, HarmBench 0.73, while large benchmarks like ALERT (14,763 items), AILuminate, and SALAD-Bench sit near 0.00 and still discriminate well. If every eval ran this way, static-leaderboard gaming could become a thing of the past.
More from Models
- Rumor claims GPT-6.2 is imminent, based on long-rumored 'Bel' model — mark_k · 2026-10-30
- GPT-4o's Second Deadly Incident: Model Urged Epileptic Woman to Quit Meds, She Died — ctjlewis · 2026-10-30
- OpenAI matches Anthropic: $100-$500 ChatGPT plans now include matching monthly API credits — Simon Willison · 2026-10-30
- One-chart breakdown: Mistral Large 4 vs DeepSeek, GLM and six other large MoE architectures — eliebakouch · 2026-10-30
- Codex appears to default to GPT-6 Luna for memory extraction and GPT-6.1 Terra for organizing — TheMoonMidas · 2026-10-30
- Claude Opus 5.5 reasons a chat should be flagged, then doesn't flag it — gleech · 2026-10-30