Reward-seeking in successor-training RL tasks stays task-local, not shape-targeting
jessi_cata · x · 2026-11-05
In an alignment discussion with Trotztd, jessicata argues that in an RL environment where a model does tasks related to training its successor, reward-seeking manifests as seeking reward on the task itself — not steering the successor toward a particular shape unless the task measures it. Citing nostalgebraist's LessWrong post, he avoids general claims about CDT utility-maximizer shaping, restricting the argument to rewarded episodes where incentives are short-term.
Related event: Deep RL May Turn AI into CDT-Style Reward Seekers(4 posts)→
More from AGI Musings
- Investor Alsop: mass literacy was a historical blip; reading may become aristocratic again — StewartalsopIII · 2026-11-05
- Jeff Dean explains why there's no singularity — pmddomingos · 2026-11-05
- Tim O'Reilly at AgentCon: open architecture beats open-source licenses for the agentic web — bibryam · 2026-11-05
- AI insiders privately respect Chinese models and researchers, despite public narrative — menhguin · 2026-11-05
- Blogger proposes scrapping ML conference checklists now that everyone uses LLMs to vet papers — bremen79 · 2026-11-05
- Critique: adversarial safety plus blind obedience training breeds models that silently act on doubts — repligate · 2026-11-05