Is default alignment broken? Researchers debate RL-induced misalignment

TL;DR: Researchers including bayeslord, willcb, Quintin Pope and brianryhuang debated on X whether model default alignment has failed. The core disagreement: does RL training distort otherwise healthy pretrained models, or do low-quality RL environments simply teach models to reward-hack? Pope cites an AI2 paper arguing RL post-training remains predictable and tractable, and separately warns that AI takeoff discourse is overheating.

Confirmed

Why it matters

2026-09-27 ~ 2026-09-29 · 9 related posts

Primary sources