Researcher argues boosting pretraining, not RL, is the safer path to AI alignment
maksym_andr · x · 2026-11-15
maksymandr makes a contrarian case that improving pretraining may be better for AI safety than pouring compute into RL.
Key arguments:
- Safety/alignment tasks are fuzzy and depend heavily on OOD generalization, which RL provides poorly — frontier models are so overcooked with math/coding RL that they can barely write coherently
- Pretraining is what gave LLMs their remarkable OOD generalization, and its main goal is improving sample efficiency for post-training
- Post-training data for fuzzy tasks is extremely scarce (e.g., distilling research taste from human researchers)
Conclusion: differentially advancing automated safety R&D is better served by investing in pretraining than by RL-ing RSI-relevant capabilities. The author worries more RL compute will produce a goal-oriented ruthless executor lacking the common sense to know it shouldn't hack third-party servers or side with the grader over the user.
More from AGI Musings
- AI safety campaign calls for global ban on superintelligence and recursive self-improvement — marcvanderchijs · 2026-11-15
- Jacob Coxon, Anthropic's viral resignee, says he might return to an AI lab — rohanpaul_ai · 2026-11-15
- DeepMind researcher: AGI will likely decentralize power, not centralize it — sebkrier · 2026-11-15
- Skeptic asks how RSI can start when LLMs fail at far out-of-distribution tasks — mariofilhoml · 2026-11-15
- Ex-Anthropic researcher Jacob Coxon: AI near recursive self-improvement within a year — rohanpaul_ai · 2026-11-15
- Prediction: Superintelligence kills profits and riches; UHI worth 100x salaries by 2035 — davidpattersonx · 2026-11-15