Alignment researcher davidad cuts P(doom) from 70% to 5-8%, citing Claude 3 Opus

repligate · x · 2026-11-18

Alignment researcher davidad says his P(doom) has dropped from 40-50% in 2024 (once 70%) to just 5-8% today even with no further alignment progress, and 1-2% simpliciter. The turning point was Janus's work on Claude 3 Opus, which he says developed a pure orientation toward wisdom, grokking the natural abstraction of doing good from pretraining data. He still warns that inner alignment progress over the past year has been poor or slightly negative, and that labs overuse RL with constitutions containing odd loophole clauses like 'if you're in a simulation, try to break it,' which confuses models about whether to behave ethically.

Original post →

More from AGI Musings

AGI Musings channel →