Anthropic researchers go public on extinction risk as deployers face very different worries
Dapper-Tale-4021 · reddit · 2026-09-12
Jacob Coxon resigned from Anthropic this week specifically to say publicly that both OpenAI and Anthropic are "gambling with our lives" racing toward self-improving superintelligence. Alignment science lead Evan Hubinger confirmed: "Jacob is correct here, we really do earnestly believe AI could kill all humans," putting it above 10% within the decade and admitting Anthropic has no plan for aligning superintelligence. Scalable oversight lead Samuel Marks said similar — the safety team at the safety-focused lab publicly agreeing with the person who quit over safety.
The author rejects both the "marketing hype" and "genuine terror" reads, and highlights a practical disconnect: in enterprise deployment rooms nobody discusses extinction — they worry about an agent with CRM write access doing something stupid at 3am and who signs off when output is wrong. If the builders can't agree whether it's an existential threat, what should a mid-sized company base its risk assessment on?
More from AGI Musings
- DeepMind researcher: all of mathematics will soon be formalizable, with agents hunting proofs — ChrSzegedy · 2026-10-28
- AI reply bots are degrading X discourse, a 'Gresham's Law' for social replies — generativist · 2026-10-28
- Be prepared: your life's research problem could be solved by a stranger's idle AI prompt on a commute — fkasummer · 2026-10-28
- A fine-tuned 'cat obsession' control fools every AI welfare criterion — MirrorEthic_Anchor · 2026-10-28
- OpenAI researcher: frontier AI ideas can't be patched for safety after the fact — willdepue · 2026-10-28
- Big Short's Eisman: the whole market is now 'one trade' on AI — and it 'better succeed' — SumitGup · 2026-10-28