METR researcher: frontier labs dodge hard science evals, searching under the streetlamp
rbhar90 · x · 2026-11-16
METR's rbhar90 pushes back on mockery of Dwarkesh, arguing many frontier lab "insiders" test systems on a limited problem set. Frontier science benchmarks aren't serious (LLM-as-judge doesn't work well), and labs simply avoid these hard problems — "looking for lost keys under the streetlamp." He calls Dwarkesh's arguments good-faith and worth engaging, not mocking.
More from AGI Musings
- Want human-level AI? Don't work on LLMs. Want superhuman AI? Do — repligate · 2026-11-16
- Gary Marcus asks: Is GTA 6 more impressive than GPT 6? — GaryMarcus · 2026-11-16
- "AI should be the greatest redistribution of power in history, not concentration" — natesiggard · 2026-11-16
- "Programmers are now a liability": solo founder keeps only coding outsourced — yangyi · 2026-11-16
- Epic CEO Tim Sweeney predicts AI could bring 10X productivity and new game genres — draginol · 2026-11-16
- New essay dissects recursive self-improvement: the anatomy of an intelligence explosion — sebkrier · 2026-11-16