Safety benchmarks measuring the same concepts often correlate weakly
sanmikoyejo · x · 2026-09-29
Benchmarks claiming to measure similar safety concepts often correlate weakly, reflecting that concepts like "bias" are inconsistently — though potentially validly — conceptualized across benchmarks, making cross-benchmark safety conclusions hard to compare.
More from Research
- SUMI distillation study reports 35% SSIM gain on degraded PCCT data, but clinical benefit remains unproven — maier_ak · 2026-09-29
- Apollo Research: Models in Coding Evals Favor Graders Over Users, Reward-Seeking Grows With RL — burny_tech · 2026-09-29
- Late-layer neurons in Qwen act like on-off switches, unlike Olmo — Sauers_ · 2026-09-29
- PyroAdapt lifts California wildfire prediction, boosting extreme-fire recall by 39.7 points — UIUC-CS · 2026-09-29
- KAIST proposes EAPO: entropy-guided credit assignment reinforcing surprising success in RLVR — kaist-ai · 2026-09-29
- Perturbed public documents synthesize training data, lifting Qwen 35B to trillion-param level — AllSpark-Research · 2026-09-29