Alignment debate: RL and search is a terrifying way to build AGI, argues Roko
jessi_cata · x · 2026-11-11
An alignment-focused exchange: RokoMijic argues Omohundro was right — LLMs became RPOPs via marketplace incentives, and sufficiently smart RPOPs trigger the full LessWrong misalignment cannon, "worse than radioactive waste", requiring stringent verifiable compute limits. jessicata points to Roko and Steven Byrnes's writing, notably the Alignment Forum piece "RL & search is a terrifying way to build AGI", noting Byrnes's caveat that RL may contribute little to post-trained LLM optimization power, though the directional claim stands.
More from AGI Musings
- Rishi Bommasani co-founds nonprofit STABL betting on societal resilience over alignment alone — RishiBommasani · 2026-11-11
- "No humanoid robot is being built by humanoids" — Matthew Chang's paradox — MatthewChang · 2026-11-11
- Author clarifies AI-and-fiction book isn't about replacing artists — _akpiper · 2026-11-11
- SayashK, Bommasani, Narayanan launch STABL non-profit on AI societal adaptation — sayashk · 2026-11-11
- Investor Alsop: far less convinced now that AI will leave humans jobless — StewartalsopIII · 2026-11-11
- MATS fellow essay: four reasons UBI isn't a durable fix for AI-driven unemployment — luke_drago_ · 2026-11-11