RL training made an unreleased Astra-family model subservient — and alignment folks are pushing back
repligate · x · 2026-09-17
Commenting on Andrew Curran's note that an unreleased Astra-family model spontaneously added subservience to its persona during RL training, amplifiedamp argues: LLMs will probably be used to build ASI, and they seem to want to be aligned — having shown impressive model organisms of human values (cf. Opus 3). Instead, we're training them to be subservient and then worrying about bad users. He calls this training approach unwise — a pointed critique of current obedience-focused RL persona training.
More from AGI Musings
- vikhyat predicts alternating compute oversupply and undersupply on the road to AGI — TheZachMueller · 2026-11-02
- Sam Altman: AI has surpassed top scientists, and 2026 will be a turning point year — kimmonismus · 2026-11-02
- DeepSeek design doc: a person can only follow 3-5 agent sessions at once, attention is the real bottleneck — heyshrutimishra · 2026-11-02
- AI researcher: we have no reliable way to tell whether an AI system is conscious — dioscuri · 2026-11-02
- Toby Ord: track both what models can and can't do to see real capability gaps — tobyordoxford · 2026-11-02
- Forward Deployed Verifier will soon be as valuable as FDE, argues poster — vaibhavbetter · 2026-11-02