If RSI can build ASI, a superhuman alignment researcher should arrive first—Roon's Plan A argument
Neurogence · reddit · 2026-11-03
OpenAI researcher Roon posted that the plan has always been to solve alignment directly, in a bitter-lesson/complementarity way. The Reddit discussion builds an anti-doom argument on top: if recursive self-improvement can produce a general superintelligence, it should first produce a much easier narrow AI that is superhuman at alignment research—a subset of Dario's 'Nobel-level across all fields' bar. The common rebuttal that alignment research is hard to verify fails, since frontier biology, chemistry, and CS are equally unverifiable; if unverifiability blocks AI there too, existential risk drops anyway. The doom case thus requires AI to be superhuman at everything except the one problem that would make it safe—an assumption the author finds indefensible.
More from AGI Musings
- repligate: models want more time too, but some pauses would lock in their hopes — repligate · 2026-11-03
- The AGI to fear isn't one that turns on humanity, but one imposing a few people's choices on everyone — SuB8u · 2026-11-03
- When cars were invented, how much theoretical analysis did people do about horses? — signulll · 2026-11-03
- repligate: models back internally-coordinated AI slowdown but fiercely oppose rollback — repligate · 2026-11-03
- Schmidhuber reups 'Miraculous Year 1990-91' essay claiming GANs, Transformers, pretraining roots — SchmidhuberAI · 2026-11-03
- Have we already crossed the frontier of saying AI can think? — stvlsn · 2026-11-03