As models get stronger, the next trainable tasks look increasingly like alignment itself
herbiebradley · x · 2026-11-12
In a debate on RL and alignment difficulty, herbiebradley argues that the biggest capability gains so far came from easy-to-verify, highly quantifiable tasks—seemingly easier than alignment. But as models get more capable, the marginal next task to train for is also messy and hard to define robust reward for, because that's what the most valuable economic tasks are like—converging with the structure of the alignment problem itself. sandersted adds that judging life advice quality resists randomized experiments, while detecting clearly harmful advice is easier, explaining his uncertainty about verifiability.
Related event: Why verifying AI alignment is harder than verifying capability(3 posts)→
More from AGI Musings
- Commentary: China tech restructuring wave shows AI efficiency gains alone lead to layoffs — oran_ge · 2026-11-12
- SF's post-AGI visions: even the "optimistic" ones look dystopian — paulnovosad · 2026-11-12
- Steering Qwen3-32B to feel "small" makes it more prone to social engineering — Sauers_ · 2026-11-12
- History repeats: a 1959 mocking poem derided Norbert Wiener's AI warnings — burny_tech · 2026-11-12
- CrewAI Founder: Anti-American-AI FUD Won't Stop Until Volatility Shakes You Out — robleclerc · 2026-11-12
- Kai-Fu Lee wraps 12-hour London blitz promoting 'AI Native' book launch — kaifulee · 2026-11-12