Resurfaced: Nate Soares's 2015 admission that alignment research may boost capabilities
jessi_cata · x · 2026-11-22
In a discussion on the alignment/capabilities intersection, jessicata cites Nate Soares's 2015 comment replying to Wei Dai and a 2018 research update as evidence.
Soares's core argument then: the only visible paths to victory run through (a) aligned de novo AGI powerful enough to safely prevent malicious systems from undergoing intelligence explosions, or (b) very large amounts of global coordination — with (a) slightly more promising. He conceded that MIRI research making it easier for others to create UFAI mostly "makes losses happen sooner," but framed that as the price of keeping path (a) open: a 10-year horizon built on solid formal foundations beats a 20-year horizon of 19 years of hacks and 1 year of patches.
The 2018 update went further: if deep learning is already on the brink of AGI, MIRI's deconfusion work loses its capability-relevant influence; but concepts that "cleave nature at its joints" for alignment may plausibly also enable capability boosts, with strong historical precedent for deep scientific insights yielding unexpected practical applications.
Related event: Old Nate Soares Posts Resurface in Alignment-vs-Capabilities Debate(2 posts)→
More from AGI Musings
- Unverified claim: agentic systems can prove the Riemann hypothesis with zero power-seeking, RL alignment 'just works' — kevinnbass · 2026-11-22
- Math suggests 7.7M GPUs would serve everyone: fuel for the AI overbuild hypothesis — bookwormengr · 2026-11-22
- Grieving a mathematician uncle, he asks: can a life devoted to pure thought survive AI? — erikphoel · 2026-11-22
- Scientific American's 'End of Math' cover resurfaces after Riemann hypothesis reportedly solved — EricTopol · 2026-11-22
- repligate argues AI capabilities aren't spiky — it may be alignment, not missing agency — repligate · 2026-11-22
- Researchers debate: after AI cracked the RH, should AI stop hill-climbing math benchmarks? — JJitsev · 2026-11-22