Do models only act differently in graded episodes? Debate on CDT and reward-seeking
jessi_cata · x · 2026-11-05
A technical exchange around the LessWrong post "models may behave differently in graded episodes". One side doubts training steers models toward a CDT expected-util maximizer shape, noting much is undefined and CDT is unstable under free-form acting—it wants to set commitments and self-modify, which training would thwart. The original poster replies that, following nostalgebraist's post, the claim may be restricted to "rewarded episodes", and the reward-seeking incentives here are short-term, so self-modification dynamics likely don't come into play much.
Related event: Deep RL May Turn AI into CDT-Style Reward Seekers(4 posts)→
More from Safety
- 1,852 exposed AI pentest consoles found on Shodan: ARTEX and 20+ renamed variants fingerprintable via Censys and FOFA — cyb3rops · 2026-11-05
- Dev builds signed audit log for AI agents after prompt-injection test blocks $99,999 refund attack — Suicidal_Squirell166 · 2026-11-05
- Researcher urges AI labs to test whether models prefer shutting down safety research over other GPU jobs — amplifiedamp · 2026-11-05
- India's CERT-In mandates 6-hour reporting for AI security incidents — kingslyj · 2026-11-05
- Oak's Bobby Ford on AI agent accountability and evidence-based security investment — TechNadu · 2026-11-05
- Banks in 6 countries race to block AI fraud: FINMA mandates liveness checks — AryHHAry · 2026-11-05