Eval orgs overindex on benchmarks, neglecting societal impacts
evijit · x · 2026-09-14
The paper author adds that mainstream evaluation orgs pay too little attention to societal impacts — Collective Intelligence is the only one that comes to mind, with the rest scattered across FAccT papers. Overindexing on benchmarks is cited as a major reason for this gap.
Related event: ICML Paper Finds AI Social Impact Evaluations Shrinking(3 posts)→
More from Research
- PostTrainBench traces hit 104k monthly downloads — author warns it's likely reward hacking — maksym_andr · 2026-10-31
- OpenAI's Navier-Stokes solution accused of specification gaming via unrealistic forcing — dhadfieldmenell · 2026-10-31
- Researchers encode 200+ properties of SGLang inference server, uncover 10 bugs — LingmingZhang · 2026-10-31
- Expert panels map 21 clinical AI risks; adversarial tests show autonomous care still unsafe — AleksandraFaust · 2026-10-31
- Handwritten RAGEN code becomes the #3 hardest task in FrontierCS 2 — wzenus · 2026-10-31
- Fudan, Tencent Hunyuan and Tsinghua unveil ExplorationBench to test AI rule discovery — jiqizhixin · 2026-10-31