LMMs-Lab's deep dive: why a +2 leaderboard score may mean better exam fit, not better ability

liuziwei7 · x · 2026-11-18

LMMs-Lab launches a new column with a long-form piece on the hard road to real evaluation. Using a period × lifecycle-stage map, it traces four eras of evaluation (borrowed CV/NLP metrics → task-specific metrics → dynamic/online eval → in-environment interaction eval) and what pretrain through deploy stages each need to answer. Key arguments: leaderboards have entered the commercial loop; runtime leakage, grader taste, and scoring policies distort scores; a +2 gain may reflect exam fit rather than ability; dynamic eval reduces one risk but adds measurement cost and isn't immune to gaming. Proposed compromise: stable core, tail probes, and real regression tests — make scores track capability, not shortcuts.

Original post →

More from Research

Research channel →