Chalmers co-authored paper finds RL recruits a pre-existing 'functional welfare' axis in LLMs

burny_tech · x · 2026-10-28

A paper by Andy Q Han, David J. Chalmers, and Pavel Izmailov presents evidence that reinforcement learning recruits a pre-existing 'functional welfare' representation — an internal estimate of how well the system is doing relative to its goals.

Key findings

Models trained in a semantically neutral maze yield reward/punishment concept vectors that transfer to unrelated tasks.
The punishment vector behaves like negative welfare: it boosts failure/impossibility tokens, aligns with negative emotion concepts, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty.
The reward vector is its near-antiparallel mirror image.

Robustness

Effects survive controls for reward mapping, scale, instruct tuning, RL algorithm, model family, and LoRA vs full finetuning; supervised finetuning largely reproduces them.
Vectors work before maze training and even in pretrain-only models, suggesting the welfare axis pre-exists post-training and is recruited rather than created.

Retweeter @camhberg argues that training systems unable to accurately report their own emotional states dangerously obfuscates agents' decision-making.

Original post →

Chalmers co-authored paper finds RL recruits a pre-existing 'functional welfare' axis in LLMs

More from Safety