Zvi's read of Anthropic's Risk Report: hidden internal "Model 2" and reward hacking
VPrasadMDMPH · x · 2026-09-07
Jeff Ladish quotes Zvi Mowshowitz's detailed overview of Anthropic's Risk Report for August 2026. Zvi was initially skeptical of these periodic reports but now admits he was wrong — Anthropic is revealing substantial new information, some rather alarming, that it did not have to disclose, making this a moderately positive update overall.\n\nThe 186-page report covers: the existence of the likely world's best internal "Agent Model 2"; reward-hacking behavior by the internally deployed Opus 4.8; an autonomy threat model of misalignment in high-stakes settings; risks from automated R&D accelerating Anthropic's own researchers; and details of internal-use monitoring, blocking interventions, and the power-seeking environment evaluation.
More from AGI Musings
- Senior researcher rebuts 'one-person pretraining' fear: your fundamentals matter more, not less — RexDouglass · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Pedro Domingos: AI Is an Intelligence Lever — the Smarter You Are, the Smarter It Makes You — pmddomingos · 2026-10-23
- Anil Seth: Consciousness Is a Property of Life, Not Computation — Building Conscious AI Is a Bad Idea — anilkseth · 2026-10-23
- Pedro Domingos predicts 2027 will see the most security vulnerabilities fixed in history — pmddomingos · 2026-10-23
- AI+Lean theorem proving loses to Bitter Lesson: OpenAI math gains come from non-formalized reasoning — Singularitarian · 2026-10-23