Should the most safety-minded lab stop working first? An AI doom strategy debate
akbirthko · x · 2026-11-23
A strategy debate around AI x-risk: the quoted post argues that if you accept the current trajectory leads to doom and an international treaty is the only fix, it seems strategically better for the most safety-pilled people (like Anthropic) to stop working last, not first. The author builds on this with a broader risk argument: invoking Piketty's r>g to warn that AI-driven monopolization of understanding would 'compress the 21st century' without the consent of 8 billion humans — a Hayekian knowledge problem. They characterize autoregressive transformers as 'cached checks so search doesn't have to terminate,' contrasted with human values as basins in a dynamical system embedded in the lifeworld, calling this the defining conflict of our era. Links to the Connectionism blog and Richard Ngo's writing.
More from AGI Musings
- AI Writing Wins Contests: Bad Models, Bad Writers, or Bad Readers? — TuhinChakr · 2026-11-23
- The New Yorker: A.I. Is Solving Decades-Old Math Problems, With Terence Tao as Guide — EricTopol · 2026-11-23
- Perelman solving Poincaré is the math equivalent of Lee Sedol taking a game off AlphaGo — gordic_aleksa · 2026-11-23
- Mathematician proposes making local models great at Lean theorem proving — StefanoGogioso · 2026-11-23
- birchlse: doubting LLMs was reasonable — the real test is whether you adapt or keep denying — birchlse · 2026-11-23
- 'AI doomers are going batshit crazy': mark_k flags threatening rhetoric from AI-risk camp — mark_k · 2026-11-23