Nat Lambert calls out labs for publishing eval scores on models nobody can use
natolambert · x · 2026-10-29
HF researcher Nat Lambert responded to Google's latest model announcement by noting that every time he wants to try a Gemini model, it isn't publicly available: "Don't bother announcing your scores if we can't use it."
He added that the timing being tied to earnings "is sus," but conceded "other labs deserve some frustration here too" — pointing out that benchmark-bragging on unavailable models is an industry-wide issue, not just Google's.
Related event: Researcher Slams Google for Benchmarking Unreleased Gemini Models(3 posts)→
More from Models
- Lie detector compiled into Qwen3.5 weights flags 91% of lies with zero false positives — teortaxesTex · 2026-10-29
- Practitioner: No serious lab actually trains on the test set today — nrehiew_ · 2026-10-29
- Benchmaxxing is mostly training focus, not cheating, argues nrehiew_ on FrontierCode — nrehiew_ · 2026-10-29
- Rumor: OpenAI pushes big release to tomorrow, possibly due to Gemini 4 Carbon — mark_k · 2026-10-29
- Gemini 4.0 Carbon's security praised, but mocked for losing the "felony bench" — vitaliychiley · 2026-10-29
- Google hits frontier on private benchmarks yet stock plunges 8% after hours — bookwormengr · 2026-10-29