Yoav Goldberg: every popular benchmark will be brute-forced, without real transfer
herbiebradley · x · 2026-09-04
Researcher Yoav Goldberg lays out a skeptical framework for measuring AI progress: any task with reliable scoring can be solved by brute-forcing training with enough money, every challenging popular benchmark meets that bar, and abilities won't necessarily transfer — so how do we measure real progress? Replies raise human-in-the-loop / centaur evals as one possible answer.
More from AGI Musings
- Consciousness Experts Clash in Nature Over Whether AI Could Ever Be Conscious — anilkseth · 2026-10-21
- Polymarket puts just 7% odds on OpenAI pausing AI training this year — Polymarket · 2026-10-21
- AI agents are overwhelming Gmail's spam filter, researcher warns — LuizaJarovsky · 2026-10-21
- Daniel Lemire: AI-generated papers are exposing academia's 'Profzi scheme' — lemire · 2026-10-21
- Universities block thesis downloads while each dissertation averages barely a handful of lifetime reads — generativist · 2026-10-21
- Berkeley wins Schmidt Sciences HAVI grant to decode whale and parrot communication with AI — begusgasper · 2026-10-21