Nathan Lambert: open models' demise exaggerated, but AA index flattered them
natolambert · x · 2026-10-28
Nathan Lambert reviews Artificial Analysis v5 and the benchmarking industry: claims of open models' demise are exaggerated, but the previous AA index did flatter them. His core argument: making benchmark scores go up has a clear, expensive playbook, while creating meaningful benchmarks at the true frontier remains under-appreciated as a hard problem — the industry needs more specific intelligence indices since LLMs serve vastly different tasks.
Related event: Nathan Lambert on the Benchmarker's Dilemma and Open Model Rankings(4 posts)→
More from Models
- CATCH: ~200-example adversarial training makes LLMs honest, beating 50× larger SFT — Mila_Quebec · 2026-10-28
- Google open-sources EmbeddingGemma 2: a 744M multimodal embedding model beating models 12x larger — dair_ai · 2026-10-28
- Users find 6.1 Terra 'lazy': it reports back every 5 minutes instead of finishing — Captain-Overboard · 2026-10-28
- Economist: OpenAI's Non-Linear Pricing Pushes Users Into Multiple Accounts — joshgans · 2026-10-28
- ShadowBench: 19 VLMs fail at judging shadow physical consistency, only Gemini-3.1-pro beats chance — HirokatuKataoka · 2026-10-28
- Community curates Awesome GPT-6.1 Terra list: projects, prompts and independent evals — Awacer · 2026-10-28