Nathan Lambert: private benchmarks show open models behind, but lack community bug-squashing
natolambert · x · 2026-10-28
Hugging Face researcher Nathan Lambert weighs in on the industry's shift toward private benchmarks: they often show open models further behind, but private benchmarks lose the community that squashes the inevitable benchmark bugs. He's 50/50 on whether private or open-source evaluation is the better approach.
Related event: Nathan Lambert on the Benchmarker's Dilemma and Open Model Rankings(4 posts)→
More from Research
- RWE-Bench launches to eval AI on real-world engineering: rockets, engines, chips — Jsevillamol · 2026-10-28
- Ben Lorica Changes His Mind on World Models as Agents Start Writing Robot Code — bigdata · 2026-10-28
- Neocambrian's from-scratch NeoSLAM tops LaMAria Long category among mono+IMU methods — AashaySachdeva · 2026-10-28
- PyNNDescent 0.7 cuts kNN-graph build times roughly in half on large datasets — leland_mcinnes · 2026-10-28
- Paper: AI and the Enclosure of the Scientific Commons — allpossiblepaths · 2026-10-28
- OPSD Paper: Self-Distillation Gains Don't Need Correct Reference Solutions — None Is Best (+2.2pp) — HanGuo97 · 2026-10-28