Clashing intuitions on alignment plausibility, illustrated via differing causal models
davidmanheim · x · 2026-11-04
davidmanheim published a long-form explainer mapping the core disagreement between "Yudkowskian AI-doomers" and "AI-whisperers" over how capabilities will evolve.
- The crux: repligate and others argue alignment is going so well that doomers should update heavily from current jagged capabilities; davidmanheim argues whether that criticism holds depends on specific capability-trajectory models.
- He illustrates both sides' intuitions with different causal models, showing how RL usage determines what updates current capability evidence should trigger.
- The optimistic case: current-level "effective problem-solving agency" did not override human values, and human-level capacity seems to retain the (previously unexpected) entanglement of capabilities and good behavior, as seen in "emergent misalignment".
- The post tags Zvi, Owain Evans, repligate and other active alignment researchers — a substantive in-community debate.
More from AGI Musings
- Should AGI labs have a reverse Conway's Law — design the org from the artifact? — akbirthko · 2026-11-04
- Verification Is Not Knowledge: Science needs more than AI answers — ersinesen · 2026-11-04
- Andreessen: Agents Are Entering Enterprises as Chaos, Just Like PCs and Spreadsheets Did — Dan_Jeffries1 · 2026-11-04
- 1000+ mathematicians sign pledge to shun AI companies and models — Shesawthat · 2026-11-04
- Von Neumann meme: "You don't have to be responsible for the world that you're in" — akbirthko · 2026-11-04
- Joscha Bach on the dream chocolate that stopped being real — Machine Learning Street Talk · 2026-11-04