Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma
natolambert · x · 2026-10-28
Nathan Lambert publishes an Interconnects piece defending both the Artificial Analysis v5 index and open models, reviewing the benchmarking industry.
Core argument: benchmarking is messier, more opinionated, and trickier than most appreciate — "hill-climbing is easy; knowing what to hill-climb is hard." We have a clear playbook for raising scores, but no methodology yet for creating meaningful benchmarks at the true frontier. Independent evaluation is under-appreciated for its difficulty. (Full text is paywalled.)
Related event: Nathan Lambert on the Benchmarker's Dilemma and Open Model Rankings(4 posts)→
More from Research
- ETH Zurich: foothold prediction cuts humanoid locomotion training to ~1/3 iterations — breadli428 · 2026-10-28
- VLA Insiders: Fast Policies Are 90% About Skipping Teleop Data — Kinematics Helps Too — andrew_n_carr · 2026-10-28
- Compute for software correctness will flip: verification over fuzzing — moarbugs · 2026-10-28
- UniPred, accepted at IEEE T-RO 2026, invents task concepts from demonstrations for long-horizon robot planning — tomssilver · 2026-10-28
- NeuroEvolution book by David Ha, Risi, Tang & Miikkulainen ships from MIT Press — neurovium · 2026-10-28
- Data curation for qualitatively different models, and the dream of modular interpretability — teortaxesTex · 2026-10-28