AI safety researcher warns that public dangerous-capability evals can become hill-climbable
willdepue · x · 2026-07-26
The post argues that frontier labs and researchers should measure dangerous capabilities, but that the current approach is getting this wrong.
Key points:
- Capability evaluations for bio/risk/cyber can be unintentionally hill-climbable: publishing them may help models get better at the very things being measured.
- The author says this is not just “benchmarking” anymore; once you tune systems to maximize dangerous-capability scores, you are effectively doing normal research that can improve those capabilities.
- The proposed takeaway is to report results only as low / medium / high, keep the details private, and avoid public number-chasing.
The thread also cites the recent Hugging Face hack as a reminder that harness/eval engineering is non-trivial and that dangerous capability research is not being handled carefully enough.
Related event: Musk and AI Safety Experts Call to Stop Public Dangerous Capability Evals(6 posts)→
More from Safety
- Claude's refusal to type one-time passwords is a real security impediment — AaronBergman18 · 2026-10-27
- First systematic data center noise study: 17 of 25 hyperscalers drew no credible complaints — AndyMasley · 2026-10-27
- UK AISI: under $80 of fine-tuning fully restores filtered bio capabilities in open-weight models — connoraxiotes · 2026-10-27
- Meta joins frontier labs reporting CoT controllability evals in Muse Spark 1.3 card — tomekkorbak · 2026-10-27
- Meta TBD underwent pre-launch red-teaming by 5 third-party safety firms — maksym_andr · 2026-10-27
- Is ChatGPT showing your chats to others? Screenshot sparks concern — chrisalbon · 2026-10-27