LLM goals can be verbalized in English before execution and don't persist across steps, study shows

davidbau · x · 2026-11-20

David Bau summarizes two surprises from Arnab Sen Sharma's goal research: (1) a sparse set of attention heads is causal for the model's next goal, and patching the signal into a verbalizer prompt makes the LM describe its upcoming action in English before execution; (2) models don't carry latent goals forward — each step re-reads the task and recomputes, so swapping another prompt's cached KV states makes it implement that prompt's sub-goal 94% of the time. Goals staying anchored in readable context is good news for monitoring prompts, CoT, and verbalizing goals directly.

Related event: MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads(7 posts)→

Original post →

More from Safety

Safety channel →