Decision Index 0.4: 224 models scored on private benchmarks only to curb benchmaxxing
multimodalart · x · 2026-10-22
multimodalart shipped Decision Index 0.4, an evaluation project designed to curb benchmaxxing:
- Scores are now built exclusively from private & in-house benchmarks
- Stronger private benchmarks on new domains
- 224 models evaluated, with new top models in every size class
The project aims to counter overfitting to public leaderboards; readers can explore it online.
Related event: Decision Index 0.4 switches to private benchmarks to fight benchmark gaming(2 posts)→
More from Models
- Nemotron 3.5 Super VL: NVFP4 on one Blackwell GPU beats BF16 on two, ~2.2x per-GPU throughput — TheZachMueller · 2026-10-22
- LFM2.5-24B-A2B Tested and Found Lacking: Looping Outputs, Broken Code, Full Configs Posted — tracagnotto · 2026-10-22
- Codex is painfully slow, Claude over-refuses: one developer stuck between both — not_rian · 2026-10-22
- I ran 40 PRs through Opus, Sonnet and GPT-6: only 11 of 37 bugs were caught by all three — ParticularFan1122 · 2026-10-22
- Whatever happened to the Yi family of LLMs? Nostalgia for 01.AI's models — code_star · 2026-10-22
- Researcher uses Grok to surface 227 suspected UAE bot accounts overnight — FlorianGallwitz · 2026-10-22