Kaj Sotala argues alignment should favor wise-advisor designs over untested CEV
xuenay · x · 2026-09-09
In a debate with Raemon about alignment approaches, researcher Kaj Sotala lays out his stance: be careful and stick to minimal solutions close to what's known to work. He argues we don't know how to build a superintelligence that would stick to a wise advisor role, but we do roughly know what wise advisors are and should be like—making it a safer baseline than speculative schemes.
Related event: Debate over CEV as Alignment Target: Sotala vs Raemon(6 posts)→
More from AGI Musings
- Benjamin Todd: AI progress beat superforecasters over 18 months — ben_j_todd · 2026-10-26
- AI researchers guessed robots wouldn't outrun humans until 2034 — it already happened — ben_j_todd · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- Anthropic: junior tech hires fell from 27% to 9% as agent hours hit 6x employee hours — thione · 2026-10-26
- Anthropic's Jascha: you may have only 1–5 years of intellectual contribution before AGI — GabGarrett · 2026-10-26