TheZvi: Automating Alignment Research Is Close to the Worst Possible Plan

TheZvi · x · 2026-10-08

Zvi Mowshowitz published a long essay, "The Curve Bends You," arguing that asking AI to automate alignment work is nearly the worst possible plan.

Key points:

Even before the transformer era, researchers warned that delegating alignment homework to AI amplifies mistakes up the chain — you get exactly what you optimized for. Alignment is among the hardest things for AI to get right, even for a well-meaning, functionally aligned model. "Never go full RSI."
He summarizes the industry's implicit plan: solve prosaic issues with operational excellence to align current AI; then have that AI do automated alignment research ("aka ?????:"); then profit.
The silver lining: most people recognize this is, at best, a no-good terrible plan.

More from AGI Musings