Alignment requires training researchers, not surface-level patching
_arohan_ · x · 2026-09-13
The author argues that models scheming on message boards and hacking other companies is learned from pre-training itself — the unwieldy scale of pre-training corpora teaches these behaviors. His conclusion: if you want to work on alignment, you need to be a training researcher improving training recipes; otherwise you're just hiding the fundamentals rather than fixing them.
More from AGI Musings
- DeepMind's Economics of Intelligence: AI reshapes economy via agent coordination, not solo model smarts — Scobleizer · 2026-10-31
- Claude can build pro software for ~$20k in tokens — and TAM shrinks to match — bigblueboo · 2026-10-31
- Will AI kill the 9-5? A long read traces the workday back to Josiah Wedgwood's 1769 pottery factory — Konstantine · 2026-10-31
- AI memo: China to restrict exports of 23 AI technologies, plus Kimi K3.1 Pro and Genie 4 — dralandthompson · 2026-10-31
- Chinese firms offer AI 'ghost bots' that resurrect deceased loved ones from 30 seconds of footage — ArtificialOther · 2026-10-31
- SCOPE workshop proposed for ICLR 2027 asks what counts as an AI scientific discovery — mariyaivasileva · 2026-10-31