The "Y-axis problem": most AI benchmarks use pretty fake units
JacquesThibs · x · 2026-11-14
LizkaVaintrob's post argues benchmarks can't tell us as much about AI progress as we'd like, largely because most have "pretty fake units." She calls this the "Y-axis problem" and discusses what to use instead and when. (Retweeted by JacquesThibs.)
More from Research
- SGLang Summit talk details battle-tested NVFP4 4-bit RL across thousands of GPUs — ericzelikman · 2026-11-14
- Decision models tested: training buys speed and cost, not reasoning back — someone_12443 · 2026-11-14
- Ropedia open-sources Xperience 2: 1,589 3D scene scans and 131 hours of egocentric demos for physical AI — liuziwei7 · 2026-11-14
- Open-Source Self-Driving Textbook on GitHub Builds an Autonomous System Chapter by Chapter — 4310sy · 2026-11-14
- RobotAgentBench: hybrid Codex-InternVLA-A1.5 system averages 48.5 on household agent tasks — jiqizhixin · 2026-11-14
- NetHack transpiler contest: serteal first to hit 100% on all public test sessions — davidbau · 2026-11-14