Benchmark indexes rotate constantly due to Goodharting, and mostly serve as lab marketing
llm_wizard · x · 2026-11-09
llmwizard explains that "index" refers to benchmark collections like the AI Intelligence Index — the sets of benchmarks people chase and watch. He notes these collections rotate benchmarks in and out constantly, largely because of Goodharting, echoing Hamel Husain's observation that most benchmarks are treated primarily as a marketing process by labs.
More from Models
- François Fleuret teases mystery model: 'v1.3 is dead, v1.4 was silly, glory to v1.5!' — francoisfleuret · 2026-11-09
- ValsAI reports critical safety flaw in frontier lab models, may delay releases — koltregaskes · 2026-11-09
- Tesla Optimus engineer builds GROXIE, a countertop robot assistant powered by Grok — Scobleizer · 2026-11-09
- Mistral Large 4 lands: 1.05T-param MoE, best Western open weights but only mid-pack — maier_ak · 2026-11-09
- ChatGPT-User crawls drop 81% as OpenAI shifts to its own index — gaganghotra_ · 2026-11-09
- LLMs show "insane regression" in intelligence when discussing cell therapy, says blogger — owl_posting · 2026-11-09