LMMs-Lab's deep dive: why a +2 leaderboard score may mean better exam fit, not better ability
liuziwei7 · x · 2026-11-18
LMMs-Lab launches a new column with a long-form piece on the hard road to real evaluation. Using a period × lifecycle-stage map, it traces four eras of evaluation (borrowed CV/NLP metrics → task-specific metrics → dynamic/online eval → in-environment interaction eval) and what pretrain through deploy stages each need to answer. Key arguments: leaderboards have entered the commercial loop; runtime leakage, grader taste, and scoring policies distort scores; a +2 gain may reflect exam fit rather than ability; dynamic eval reduces one risk but adds measurement cost and isn't immune to gaming. Proposed compromise: stable core, tail probes, and real regression tests — make scores track capability, not shortcuts.
More from Research
- MIT's Markus Buehler to receive 2027 SES medal for pioneering physics-aware AI — ProfBuehlerMIT · 2026-11-18
- Field experiment: Google AI Overviews cut publisher clicks 39.8% with no UX gain — paulnovosad · 2026-11-18
- RoboDojo sim "not even wrong": blog exposes physics flaws in popular robot benchmark — ZeYanjie · 2026-11-18
- CoTAM: Machine-Oriented Image Compression That Preserves Multimodal AI Reasoning Performance — burkov · 2026-11-18
- Formal verification meets agentic AI: neurosymbolic approach pitched as a path to trustworthy agents — luislamb · 2026-11-18
- Claude correctly folds ubiquitin from a plain prompt, no specialized tools needed — iScienceLuvr · 2026-11-18