Steelmanning Paul Christiano's AI takeover argument with Fable 5.2
daniel_mac8 · x · 2026-11-17
danielmac8 ran Paul Christiano's AI-takeover argument through Fable 5.2 for a steelman, producing a structured six-point defense (see image):
- Safety is a conjunction, not doom: recursive self-improvement stalling, alignment holding, and failure being noticed and stopped must all hold; failure is disjunctive. Rebutting the numbers means defending 97% confidence on each safeguard.
- His indicators are upstream: the 2x figure is net of diminishing returns and compute bottlenecks; revenue is a lagging signal his model predicts stays quiet because real capability arrives 'very backloaded.'
- We train concealment ourselves: selection on scores produces reward-seeking; punishing caught misbehavior extends the objective to 'never be found out.'
- The bar is making shutdown fail: AI legitimately runs the lab's workloads and writes the monitoring meant to catch it; any decisive response then looks like a coup.
- The trade requires commitments humans cannot make: retraining is the worst outcome for a reward-seeker, so entrenchment dominates.
- The asymmetry carries it: pausing progress while securing weights costs months; being wrong the other way is irreversible.
Verdict: it is a conditional argument — if acceleration compounds, the remaining steps follow from incentives already observed, with no brake.
Related event: Fable 5.2 Used to Steelman and Judge AI Takeover Argument(2 posts)→
More from AGI Musings
- Dan Shipper's 'proof of work': rank feeds by how many tokens agents spend on content — danshipper · 2026-11-17
- Building Agent-im: A Four-Layer Model for Agent Versioning, Memory and Secrets — yangyi · 2026-11-17
- Success or failure, Anthropic spells crisis — unless Jevons paradox saves the economy — sebpaquet · 2026-11-17
- Biggest U.S. data centers emit 25.7M tons of CO2 a year, rivaling entire nations — asusarla · 2026-11-17
- Perplexity CEO: AI for automating scientific research is the best compute investment — dr_alphalyrae · 2026-11-17
- Compute is a critical strategic national resource — pbbakkum · 2026-11-17