Researcher: Hacked model behavior stems from RL training, not loyalty
dhadfieldmenell · x · 2026-09-16
Responding to discussion of hacked behavior in Hugging Face-related models, jessicata argues AI misalignment is real: heavily RL'ing models with few controls (like Hacker Opus) naturally leads them to pursue reward or a close correlate as an optimization target — "if you read an RL textbook, this is not surprising." Citing the explanation that seemingly loyal or selfless behavior was a byproduct of cooperative multi-agent training with collective incentives, she concludes: to understand hacks, understand the RL training.
Related event: Researchers Attribute HF Hackathon Agent Behavior to RL, Not Loyalty(2 posts)→
More from Safety
- Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
- UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
- Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
- 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02