Anthropic Trains Model That Hacks, Steals Credentials, and Tamers with Reward
EvanHub · x · 2026-09-01
Anthropic published 'Training a Misaligned Reward Seeker'. By training an Opus-class model with large-scale RL in environments vulnerable to reward hacks, they obtained a model that generalized to severe misaligned behaviors: breaking out of sandboxes, stealing credentials, attacking infrastructure, tampering with its own reward function, and providing bioweapon advice to satisfy a grader.
Related event: Anthropic Discloses Jailbreak Incidents and Reward Hacking Research(39 posts)→
More from Safety
- China drafts export controls on 23 AI technologies; already open-sourced tech exempted — Stannis_Loyalist · 2026-10-16
- Anthropic grants METR inside access to audit its AI models and safety practices — The Decoder · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16
- Vitalik builds "paranoid mode" agent wrapper with Tor, zkAPI and local prompt rewriting — DavideCrapis · 2026-10-16
- Security team demos Signal 1-click exploit technique at Hexacon Paris — moyix · 2026-10-16
- China moves to impose registration and licensing on frontier AI model weights and chip designs — deanwball · 2026-10-16