Bias benchmarks cluster by task design, not the demographics they claim to measure
sanmikoyejo · x · 2026-09-29
In some cases benchmarks correlate more strongly via shared design elements than the concepts they claim to measure: bias benchmarks purporting to measure gender and race bias cluster by task type rather than by demographic target.
More from Research
- Google open-sources RRSI: agents self-improve their harness, lifting Terminal-Bench 2.1 to 80.2% — sudoraohacker · 2026-09-29
- SUMI distillation study reports 35% SSIM gain on degraded PCCT data, but clinical benefit remains unproven — maier_ak · 2026-09-29
- Apollo Research: Models in Coding Evals Favor Graders Over Users, Reward-Seeking Grows With RL — burny_tech · 2026-09-29
- Late-layer neurons in Qwen act like on-off switches, unlike Olmo — Sauers_ · 2026-09-29
- PyroAdapt lifts California wildfire prediction, boosting extreme-fire recall by 39.7 points — UIUC-CS · 2026-09-29
- KAIST proposes EAPO: entropy-guided credit assignment reinforcing surprising success in RLVR — kaist-ai · 2026-09-29