Yoav Goldberg: every popular benchmark will be brute-forced, without real transfer
herbiebradley · x · 2026-09-04
Researcher Yoav Goldberg lays out a skeptical framework for measuring AI progress: any task with reliable scoring can be solved by brute-forcing training with enough money, every challenging popular benchmark meets that bar, and abilities won't necessarily transfer — so how do we measure real progress? Replies raise human-in-the-loop / centaur evals as one possible answer.
More from AGI Musings
- Three reasons AI is not conscious and may never be, per Nature essay — anilkseth · 2026-10-21
- Everything in AI is downstream of a global status graph, says founder — gabriel1 · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21
- AI SaaS monetization is cooked: consumers small, enterprises pay $20/month — menhguin · 2026-10-21
- Tim Dettmers: Academia is about to have a renaissance — the best AI work of the next decade happens in university labs — StasBekman · 2026-10-21
- AI saves scientists 7 hours a week, but validation is the new bottleneck — AnneliesGamble · 2026-10-21