Is alignment just reward shaping? A debate on RL alignment strategies
JacksonKernion · x · 2026-10-13
tszzl argues current alignment is all reward shaping of LM RL — sometimes via constitutions or automated graders — and calls newer methods like alignment SDFs unpredictable alchemy. Jackson Kernion pushes back: show me a misaligned behavior and he can show a reward that would have prevented it; most production misalignments look like oversights.
More from AGI Musings
- Dev who made $20k in 3 days jokes he has no idea what 'Jev' is — clients still pay — aryanXmahajan · 2026-10-13
- REA hits 120k stars: when AI can reverse-engineer any software, what's left? — Arindam_1729 · 2026-10-13
- REA hits 120k stars: AI reverse-engineers any software — so what's left to defend? — AlchainHust · 2026-10-13
- IMF warns AI share crash now threatens global financial stability — nordicinst · 2026-10-13
- Yacine: AI makes compute a more attractive investment than rental housing — yacineMTB · 2026-10-13
- Mathematicians say models feel misaligned, and x-risk fervor may be self-fulfilling — akbirthko · 2026-10-13