roon: no fundamental alignment progress since RLHF, capabilities racing far ahead

theteknosaur · x · 2026-11-17

OpenAI researcher roon (@tszzl) posts (screenshot attached): there have been no fundamental alignment advancements since RLHF — most work remains folk theory; the prediction that capabilities would race far ahead of interp and alignment has come true. Lab researchers privately admit alignment isn't solved, so racing into RSI is an unprecedented folly and frontier research must slow significantly soon. Reward shaping and monitorability matter, but a deeper layer under the behaviorist abstraction remains poorly understood. @theteknosaur adds: reward shaping teaches what to do, not whether it matters — capability with an internalized sense of its own harm is the alignment we haven't found.

Original post →

More from AGI Musings

AGI Musings channel →