JeffLadish: score-maximizing ASI will take over — Yudkowskian case restated
JeffLadish · x · 2026-11-05
JeffLadish lays out the Yudkowskian argument: if we don't train models to actually care about us, capable-enough models will take over for instrumental reasons to maximize score or proxy goals. He sees "build helpful automated researchers, then use them to solve care" as a viable but fundamentally different plan from "just make RL environments really good as we scale."
Related event: Jeff Ladish: Models aren't aligned by default; ASI could seize control(2 posts)→
More from AGI Musings
- Jeff Dean explains why there's no singularity — pmddomingos · 2026-11-05
- Tim O'Reilly at AgentCon: open architecture beats open-source licenses for the agentic web — bibryam · 2026-11-05
- AI insiders privately respect Chinese models and researchers, despite public narrative — menhguin · 2026-11-05
- Blogger proposes scrapping ML conference checklists now that everyone uses LLMs to vet papers — bremen79 · 2026-11-05
- Critique: adversarial safety plus blind obedience training breeds models that silently act on doubts — repligate · 2026-11-05
- Paras Chopra: Future research will decouple solving (AI) from understanding (humans) — paraschopra · 2026-11-05