Rant: Gaming AI Model Benchmarks with Custom Metrics
JJitsev · x · 2026-07-13
The author rants about the current chaos in AI model evaluation: making one's own model look the strongest by inventing scoring metrics that no one else uses. For example, in the SOOFI evaluation, using a so-called "capability index" to evaluate base models made Qwen 3 32B appear weaker than practically insignificant models like Teuken.
Related event: Controversy Over Nemotron Reproduction and Custom Benchmarks(4 posts)→
More from Models
- OpenAI rolls out voice in GPT-Live, but the UI obscures search and reasoning — Graham_dePenros · 2026-07-22
- Moonshot’s Kimi K3 sets a new open-weights ECI record at 156 — scaling01 · 2026-07-22
- Nanbeige4.2-3B launches as a 3B Looped Transformer model that beats larger baselines — Wooden-Deer-1276 · 2026-07-22
- A post says six companies now beat Google’s best LLM, including two open-source models — soham_btw · 2026-07-22
- Google says Gemini 3.5 Pro is in testing and Gemini 4 is already pre-training — Wide-Ad1564 · 2026-07-22
- Gemini 3.5 Flash Lite Tested: Not Frontier-Optimal, but Hits 350 tok/s — brandon_galang · 2026-07-22