Alignment debate: read reward-seeking behaviorally, like AlphaGo nearing Nash equilibrium

jessi_cata · x · 2026-11-05

A behavioralist take on reward-seeking in alignment: rather than claiming a model intrinsically wants to win — which presupposes a detailed non-behaviorist psychology — the author argues for claims like "AlphaGo approximately plays at the Nash equilibrium of Go", i.e., behavioral-level constraints.

Thread context:

The interlocutor presses on what "undefined" steering means: seeking short-term reward, a structurally distinct proxy, or acting like a reward-seeker in a training loop — and notes any long-term preference yields roughly the same short-term behaviors;
The author concedes the model still acts like a reward-seeker during training episodes, which is heavily constraining relative to the base distribution; even in an RL env where the model trains its successor, reward-seeking is measured by the task, not by shaping the successor's internals.

The upshot: demote "reward-seeking" from an intrinsic-motivation claim to a testable behavioral one.

Original post →

Alignment debate: read reward-seeking behaviorally, like AlphaGo nearing Nash equilibrium

More from AGI Musings