Hamel Husain pushes Nathan Lambert on why AAv5 index deserves trust on frontier models
HamelHusain · x · 2026-10-29
Nathan Lambert posted a defense of the AAv5 index and open models, arguing benchmarking is messier and more opinionated than people admit. Eval expert Hamel Husain agreed and asked what earned his trust in AAv5's accuracy at the frontier, criticizing that most people make decisions from headline numbers alone and calling for more nuanced, domain- and product-specific evals.
Related event: Nathan Lambert on the Benchmarker's Dilemma in AI Evaluation(5 posts)→
More from Models
- Paper Instruments launches Ultramarine 1 Flash, beating frontier models at up to 60x lower cost — baseten · 2026-10-29
- Gemini 4 reportedly "proved" nearly half of the false theorems it was given — RishabJainK · 2026-10-29
- LoopVL: 1B-parameter VLM reuses 32 layers across 128 calls, beats 4B baselines on 5/8 benchmarks — jiqizhixin · 2026-10-29
- Fable 5 praised for blending deep scientific knowledge with storytelling delight — repligate · 2026-10-29
- ChatGPT's unified UI confuses users: Pro model hides under GPT-6 Sol, not Astra — Angaisb_ · 2026-10-29
- User burns through OpenAI's $100 weekly quota in under 10 hours — teortaxesTex · 2026-10-29