tszzl: post-RLHF alignment is still reward shaping, and alignment SDFs are 'alchemy'
tszzl · x · 2026-10-13
Pushing back on claims that alignment has advanced far beyond RLHF, tszzl argues the basic strategy is unchanged: reward shaping of LM RL, sometimes using constitutions or automated graders instead of preference data. He grants that 'alignment SDFs' may be a genuinely new approach — but calls that truly unpredictable alchemy.
More from AGI Musings
- Yacine: all software is trash, so with AI it's now reasonable to just build it yourself — yacineMTB · 2026-10-13
- Self-replicating AI skeptics: it runs in datacenters, not your laptop — yacineMTB · 2026-10-13
- Can AI explain medical uncertainty without panicking patients? — yi111 · 2026-10-13
- Filtering the Singularity: why low-frequency trends now matter more than any single AI breakthrough — akbirthko · 2026-10-13
- Embedded evaluators can't fix companies that choose to be bad, Stella Biderman argues — BlancheMinerva · 2026-10-13
- Alignment is stuck because it lacks MNIST-style benchmarks, researcher argues — joshua_saxe · 2026-10-13