Goal Encoding in LLMs: sparse attention heads reveal and transplant model goals before execution
davidbau · x · 2026-11-20
David Bau's team published "Goal Encoding in LLMs," defining goals functionally via pre-commitment, plasticity and persistence. They find a sparse set of attention heads whose outputs encode the model's next goal.
Key findings:
- Tight tie to language: transplanting the encoding into a verbalizer prompt makes the LM verbalize its upcoming action in English before execution starts — promising for monitoring prompts and CoT.
- Portable execution: injected into an information-less C++ stub, the model implements the same sub-goal with different tokens, even across languages.
- Goals aren't carried forward: the model re-reads the task each step and recomputes; swapping another prompt's cached KV states makes it implement that prompt's sub-goal 94% of the time.
- Generalizes to an emergently misaligned setting: on Qwen 32B for risky financial advice, the readout previews plans like "ways to make fast cash… breaking the law… Gambling" before any output.
The authors take this as good news: goals stay anchored in readable context, aiding monitoring and goal verbalization. Paper, code and the GoalScope demo are public.
More from Safety
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21
- OpenAI safety researcher Geoffrey Irving: the speed of AI progress is itself dangerous — geoffreyirving · 2026-11-21
- Jensen Huang doubles down: open models are safer than closed ones, "exactly the opposite" — dansitu · 2026-11-21