dlwh: every eval is broken, yet blindly hillclimbing on them drove stunning AI progress
dlwh · x · 2026-09-10
AI researcher dlwh offers a counterintuitive take: the second most unreasonably effective thing about AI is that despite virtually every eval being badly broken, blindly hillclimbing on those evals has still produced incredible progress — highlighting the paradox between flawed benchmarks and real capability gains.
More from AGI Musings
- Anthropic-aligned researcher puts >10% odds on AI killing all humans this decade, sparks 'doomsday cult' backlash — TheMoonMidas · 2026-09-10
- New AI whistleblower draws Blake Lemoine comparisons: the engineer fired for claiming AI was sentient — IsForAt · 2026-09-10
- Ex-Anthropic researcher launches Tailwind: $200M waiting for ambitious AI safety projects — jam3scampbell · 2026-09-10
- Anthropic Economics releases 2030 AI impact model; critics say scenarios aren't extreme enough — scaling01 · 2026-09-10
- Ex-Anthropic researcher's WSJ + Fox News blitz fuels claims of a coordinated campaign — GabGarrett · 2026-09-10
- MacAskill & Moorhouse: intelligence explosion could compress a century of progress into a decade — zetalyrae · 2026-09-10