Séb Krier on the Christiano/Dwarkesh podcast: bullish on capabilities, skeptical of AI takeover
sebkrier · x · 2026-11-19
Séb Krier published early reactions to Dwarkesh's podcast with Paul Christiano: bullish on rapid capability gains, skeptical of the leap to takeover, and wary of emergency overreach.
- On RSI: he argues "recursive self-improvement" is a bad initialism — better understood as automation of the model training pipeline. Automation-driven capability gains are real and desirable, even helpful for safety; he expects a relatively slow takeoff, largely manageable, and is more optimistic about society's and technology's spontaneous adaptation.
- On takeover predictions: claims like "AI that could overthrow governments within 6 months" or "fairly likely many people die" are extremely unlikely and unrealistic — the interview offers no convincing concrete mechanism for physically defeating a modern military via server intrusion or bribes.
- On AI misbehavior: the standard argument (reward motive in training transfers to deployment → motive to manipulate evaluators) still lacks persuasive force about why an AI would seek to seize power.
He stresses this is a disagreement about argument quality, not a denial that AI capabilities are improving rapidly.
More from AGI Musings
- DC Moves to Legalize Online Casinos While Keeping Waymo Banned to Protect Jobs — AndyMasley · 2026-11-19
- "If Claude has phenomenal experience, RL is hell": Anthropic IPO sparks model welfare jokes — erikphoel · 2026-11-19
- Turing Award winner Yann LeCun tells students: take quantum mechanics over iOS programming — ylecun · 2026-11-19
- Fintech founder argues trust, not capability, is the bottleneck for AI in markets — ycombinator · 2026-11-19
- Toby Ord: AI 80%-reliability horizon grew from 15 to 25 minutes, but 80-90% cap can't close the R&D loop — tobyordoxford · 2026-11-19
- VirtualBiotech: tens of thousands of AI scientist agents for drug discovery, in Science — james_y_zou · 2026-11-19