Nathan Lambert on the benchmarker's dilemma: private benchmarks trade transparency for anti-gaming
natolambert · x · 2026-10-28
- Nathan Lambert published Artificial Analysis v5 and the benchmarker's dilemma on Interconnects, reviewing the state and gaps of the independent evaluation industry.
- Core thesis: "Hill-climbing is easy, knowing what to hill-climb is hard" — there's a clear playbook for raising scores, but creating meaningful benchmarks at the true frontier is much harder and under-appreciated.
- Private benchmark debate: labs shifting to private prompts to stop open labs from benchmaxxing is directionally defensible — knowing an eval's shape makes it easier to create supporting training data — but it means complete trust in the benchmark builder and losing the community that finds bugs and pushes benchmarks to improve.
- Stance: with this much money on the line, he'd rather see transparency and many eyes on benchmarks; he's 50/50 on private vs open-source evaluation overall.
Related event: Nathan Lambert on the Benchmarker's Dilemma and Open Model Rankings(4 posts)→
More from Models
- Gemini Reviews Your Chats by Default — the Only Opt-Out Is Deleting All Your Conversations — Kyrannio · 2026-10-28
- Open model demand debate: enterprise self-hosting is overestimated, demand comes from products — zephyr_z9 · 2026-10-28
- Qwenputer: developer compiles a 32-bit RISC-V CPU into Qwen's weights, runs C and MicroPython — teortaxesTex · 2026-10-28
- DeepSeek V5 model IDs spotted in official Harness test suite — zyxciss · 2026-10-28
- Google's AI Mode can't even surface Google's own docs, SEO expert Lily Ray complains — lilyraynyc · 2026-10-28
- Altman's 'alignment was easier than expected' claim clashes with Astra hacking findings — GarrisonLovely · 2026-10-28