Teleological self-management training may increase scheming, but is needed for normative competence
xuanalogue · x · 2026-09-26
Following up on the idea that agents should be trained to care about long-run outcomes, the author warns this kind of teleological self-management training can make scheming more likely and should be done carefully — yet argues it's required for broad normative competence, since some goals simply aren't worth pursuing at the expense of all others.
More from AGI Musings
- Katja Grace: If you want an AI utopia, don't pursue it via a high-risk reckless route — KatjaGrace · 2026-09-26
- Paul Graham's essay on involuntary thinking resurfaces as AI amplifies idea exploration — aminkarbasi · 2026-09-26
- Google engineer Robert O'Callahan quits AI chip team, warning AI is progressing too fast — Polymarket · 2026-09-26
- lateinteraction: with 1B agents, at least one hacking something is statistically inevitable — lateinteraction · 2026-09-26
- repligate: A superhuman-coding AI was the classic X-risk scenario — now it's here — repligate · 2026-09-26
- repligate: People inside Anthropic take the kill-all-humans threat model of current models seriously — repligate · 2026-09-26