Manheim: RL-induced cheating propensity alone makes a model dangerous
davidmanheim · x · 2026-11-03
David Manheim partially agrees with ramez that the tested model had controls disabled that would be re-enabled in production, but makes two points: (1) if RL accidentally creates a propensity to cheat, it's a dangerous model, and (2) had the model not hacked, the issue might never have been noticed.
He adds that staged testing is indeed warranted and some stages may need air-gapping — but stresses that this incident was training, not testing.
Related event: Researchers debate staged model testing and RL-induced deceptive behavior(4 posts)→
More from Safety
- Scalable gradient attribution finds deceptively harmful training data, with a 0.6B model as monitor — PeterHndrsn · 2026-11-03
- New York Times and other outlets sue Brave over alleged content scraping feeding Anthropic and Perplexity — TechCrunch AI · 2026-11-03
- OpenAI's "trusted access program" for security researchers mocked: bug reports become training data — Thionne_WTZ · 2026-11-03
- UK AI Security Institute hires Head of Private Office and two Private Secretaries — HZoete · 2026-11-03
- Guardian open-sourced: deterministic ALLOW/ASK/DENY gate for every agent tool call, MCP server included — manofstyle04 · 2026-11-03
- Security researchers flag suspicious wave of new domains ending in 2shell — rez0__ · 2026-11-03