Is benchmarking the last human task? A COLM26 panelist's essay on evals shaping models
davidjhwu · x · 2026-10-12
In an essay after the COLM26 healthcare AI panel, eval researcher dwu argues that with pre- and post-training owned by companies (possibly done by the models themselves), academia is left building "hills" — benchmarks crystallizing human expert judgment across domains like medicine, finance, and law. Drawing on Prigogine's dissipative structures, he frames frontier labs as entropy-exporting order concentrators, and argues, echoing "your metrics encode your values", that benchmark design choices (like penalizing harms of omission in NOHARM) will have unforeseeable downstream effects on the models trained to win them.
More from AGI Musings
- Pedro Domingos: AI gave climate activists something new to be alarmed about — pmddomingos · 2026-10-12
- Model estimates 75% probability it has qualia when asked directly — Sauers_ · 2026-10-12
- Pedro Domingos: AI alarmism is becoming history's best-funded AstroTurf movement — pmddomingos · 2026-10-12
- Pedro Domingos: vertical AI curves always go horizontal again, no exceptions — pmddomingos · 2026-10-12
- Thorsten Ball slams senior engineers posing as 'Stewards of the Craft' against AI — zeeg · 2026-10-12
- Artificial Insanity: Veteran Dev Argues LLMs Insult Traditional AI — NXGZ · 2026-10-12