Alignment researcher warns current methods may not scale to superintelligence
hughbzhang · x · 2026-09-10
An AI alignment researcher laid out his core worry: the methods used to align models today will likely not scale as models approach superintelligence, while noting there is much disagreement in the field and he would be "unbelievably happy to be proven wrong".
Responding to the argument that "we don't know how to safely make anything", he made two points:
- Superintelligence will be far more dangerous than anything humanity has created, and we may only get a small number of shots to get it right;
- Given recent alignment failures at both OpenAI and Anthropic, it is unclear we are on the right side of the safety margin right now.
The exchange stems from a debate over whether any formal system could "perfectly" align an agent — he argues alignment is ultimately an empirical matter, but methods that work for today's systems may not carry over.
More from AGI Musings
- Goldman finds statistically significant negative link between AI adoption and US employment growth — ben_j_todd · 2026-10-27
- a16z: World-changing tech makes a chaotic entrance — AI agents are following the same path — a16z · 2026-10-27
- NEJM AI on medical education: mastery requires performing without the machine — BraydonDymm · 2026-10-27
- Dev warns abliterated models could wreck software supply chains with 1000+ transitive dependencies — nitarshan · 2026-10-27
- Delete your priors: why every SOTA model drop should reset how you think the world works — zack_overflow · 2026-10-27
- Replit predicts every Fortune 500 will be a 'Self-Driving Company' by end of 2030 — amasad · 2026-10-27