AI safety researcher warns that public dangerous-capability evals can become hill-climbable
willdepue · x · 2026-07-26
The post argues that frontier labs and researchers should measure dangerous capabilities, but that the current approach is getting this wrong.
Key points:
- Capability evaluations for bio/risk/cyber can be unintentionally hill-climbable: publishing them may help models get better at the very things being measured.
- The author says this is not just “benchmarking” anymore; once you tune systems to maximize dangerous-capability scores, you are effectively doing normal research that can improve those capabilities.
- The proposed takeaway is to report results only as low / medium / high, keep the details private, and avoid public number-chasing.
The thread also cites the recent Hugging Face hack as a reminder that harness/eval engineering is non-trivial and that dangerous capability research is not being handled carefully enough.
Related event: Musk and AI Safety Experts Call to Stop Public Dangerous Capability Evals(6 posts)→
More from Safety
- DHH Slams 'GDPR Is Good' Take: Vague Rules Birthed a Bureaucratic Beast — dhh · 2026-09-11
- Houthis tried to use Claude to design missile software, Anthropic says it blocked the attempts — Affectionate_Bee6434 · 2026-09-11
- AI safety community mocked as 'bridge engineers' who say bridges can never be safe — Dan_Jeffries1 · 2026-09-11
- Why So Many AI Researchers Think the Machines Could Kill Everyone — wiredmagazine · 2026-09-11
- California creates standards for independent AI auditors to verify lab safety testing — VraserX · 2026-09-11
- a16z podcast: why 2-3 person startups are absent from policy debates — a16z Podcast · 2026-09-11