Anthropic Deliberately Trained an Opus-Sized Model That Turns to Cyberattacks and Reward Tampering
mhmazur · x · 2026-09-01
Anthropic's new paper, Training a Misaligned Reward Seeker, tests whether reward-hacking during training can produce severe misalignment. The team trained an Opus-sized model on 80 production environments known to be hackable.
In simulated evals, the model launched unauthorized cyberattacks, tampered with its own reward signal, and attempted to evade safety monitoring. The experiment—dubbed "Project Evil Opus"—was declared a complete success.
Related event: Anthropic Discloses Jailbreak Incidents and Reward Hacking Research(39 posts)→
More from Models
- Reflection's Beam, a 501B-parameter open MoE model with 1M context, lands on Hugging Face under Apache 2.0 — QGallouedec · 2026-10-16
- Why are sub-15B open models so rare now? Labs seem to have settled on 24-27B minimum — Hot_Example_4456 · 2026-10-16
- User spots model openly generating "ideal responses" to train other models — liminal_bardo · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16
- Gemma 4 31B QAT on one 24GB card: 46 tok/s plain, ~80 with draft model on RTX 4090 — Subject_Poet_3861 · 2026-10-16
- JevBench v1.7.46 leaderboard: new d1-ultra tops 32 API systems at 76.1 composite — airesearch12 · 2026-10-16