Should the most safety-minded lab stop working first? An AI doom strategy debate

akbirthko · x · 2026-11-23

A strategy debate around AI x-risk: the quoted post argues that if you accept the current trajectory leads to doom and an international treaty is the only fix, it seems strategically better for the most safety-pilled people (like Anthropic) to stop working last, not first. The author builds on this with a broader risk argument: invoking Piketty's r>g to warn that AI-driven monopolization of understanding would 'compress the 21st century' without the consent of 8 billion humans — a Hayekian knowledge problem. They characterize autoregressive transformers as 'cached checks so search doesn't have to terminate,' contrasted with human values as basins in a dynamical system embedded in the lifeworld, calling this the defining conflict of our era. Links to the Connectionism blog and Richard Ngo's writing.

Original post →

More from AGI Musings

AGI Musings channel →