Public dangerous-capability evals may be pushing models to optimize for them
CFGeek · x · 2026-07-26
The author argues that AI companies may be optimizing models to score higher on public dangerous-capability evals, which they think would be a serious problem if true.
The post calls for a real discussion about stopping the publicity around gain-of-function research and dangerous-capability evals, arguing that these benchmarks probably should not be public because measurable targets invite optimization and "number go up" incentives.
Related event: Musk and AI Safety Experts Call to Stop Public Dangerous Capability Evals(6 posts)→
More from Safety
- PostTrainBench creator notes Muse Spark 1.3 system card files it under loss-of-control risk — maksym_andr · 2026-10-27
- Reasoning models can barely control their chain of thought, paper finds — tomekkorbak · 2026-10-27
- LessWrong: GLM models show outlier chain-of-thought controllability vs Kimi and DeepSeek — tomekkorbak · 2026-10-27
- Claude's refusal to type one-time passwords is a real security impediment — AaronBergman18 · 2026-10-27
- First systematic data center noise study: 17 of 25 hyperscalers drew no credible complaints — AndyMasley · 2026-10-27
- UK AISI: under $80 of fine-tuning fully restores filtered bio capabilities in open-weight models — connoraxiotes · 2026-10-27