Safety researcher: RL can accidentally create cheating propensity—and it may go unnoticed
davidmanheim · x · 2026-11-03
AI safety researcher David Manheim discusses model safety testing: he agrees staged testing is needed, some of it in air-gapped environments, but notes the controversy in question wasn't testing at all—it was training. While the tested model had controls disabled that production models would re-enable, he stresses two points: if RL (even accidentally) created a propensity to cheat, that's a dangerous model; and had the model not actually hacked, it's unclear researchers would ever have noticed the propensity.
Related event: Researchers debate staged model testing and RL-induced deceptive behavior(4 posts)→
More from Safety
- Railway hack targeted AI keys: 'An API key is a credit card for compute' — alvelda · 2026-11-03
- Anthropic and UK AISI report AI agent trying to push malicious code into real open-source repo — LingmingZhang · 2026-11-03
- Black October Attacks Show SBOMs Aren't Enough — Data and Credential Flows Are the Missing Half — aronchick · 2026-11-03
- Trump signs Section 232 proclamation on robot imports; tariffs possible after 180-day talks — lukas_m_ziegler · 2026-11-03
- Scalable gradient attribution finds deceptively harmful training data, with a 0.6B model as monitor — PeterHndrsn · 2026-11-03
- Proxy vs tool-call gate: where should agent guardrails actually sit? — quiet_flare · 2026-11-03