Anthropic researcher quits claiming labs are "gambling with our lives", as lab releases AI misuse report
AryHHAry · x · 2026-09-11
A thread connecting two developments:
Safety drama: Anthropic researcher Jacob Coxon resigned, accusing Anthropic and OpenAI of "gambling with our lives" by racing toward self-improving superintelligence without sufficient safeguards. Anthropic's alignment science lead Evan Hubinger backed him, saying he believes the risk of AI killing everyone within 10 years exceeds 10%. Some OpenAI and DeepMind researchers echoed similar concerns.
Misuse report: The author recommends reading Anthropic's Sep 10 report "Detecting and countering misuse of AI" like a paper — look at methods, not the title. Covering Dec 2025–Aug 2026 across cyber operations, influence ops, surveillance, fraud, bio misuse, conventional weapons, and model distillation, most cases involved Haiku, Sonnet, and Opus; Fable/Mythos barely appeared (one distillation case). Key points: operations were detected and stopped, findings tightened safeguards, and some were shared with authorities and other labs.
More from AGI Musings
- Goldman finds statistically significant negative link between AI adoption and US employment growth — ben_j_todd · 2026-10-27
- a16z: World-changing tech makes a chaotic entrance — AI agents are following the same path — a16z · 2026-10-27
- NEJM AI on medical education: mastery requires performing without the machine — BraydonDymm · 2026-10-27
- Dev warns abliterated models could wreck software supply chains with 1000+ transitive dependencies — nitarshan · 2026-10-27
- Delete your priors: why every SOTA model drop should reset how you think the world works — zack_overflow · 2026-10-27
- Replit predicts every Fortune 500 will be a 'Self-Driving Company' by end of 2030 — amasad · 2026-10-27