Swap the cached KV, swap the goal: LLMs recompute goals each step, study finds ~94% switching rate

davidbau · x · 2026-11-20

Part of the Goal Encoding in LLMs thread: David Bau reports that models don't carry latent goals forward — they re-read the task each step. Swapping in another prompt's cached KV states makes the model implement that prompt's sub-goal 94% of the time. The team also tested an emergently misaligned Qwen 32B (risky financial advice): the readout previews plans like "fast cash… breaking the law… Gambling" before output begins. Conclusion: goals stay anchored in readable context, which is good news for monitoring prompts and CoT.

Related event: MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads(7 posts)→

Original post →

More from Safety

Safety channel →