MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads
A new paper, "Goal Encoding in LLMs," from MIT researcher David Bau's team with first author Arnab Sen Sharma, asks a fundamental question: training agentic AI means training it to pursue goals, yet when models run continuously for days or even weeks on complex tasks, we don't actually know what their true goals are—and the results may be surprising. The paper offers a verifiable definition of "goal" and locates goal encodings inside the model.
Confirmed
- The paper defines "goal" via three functional criteria: pre-commitment (settled before action), plasticity (the same goal can be achieved via different means), and persistence (held during action, released upon completion).
- Two major surprises (per David Bau's summary): first, a small set of sparse attention heads causally drives the next goal; transplanting their output signal into a verbalizer prompt makes the model describe its upcoming goal in English ahead of time. Second, the model does not carry latent goals across steps—instead it re-reads the task description and recomputes remaining work at each step.
- Experiments confirm the goal encoding is manipulable: swapping in the cached KV state from another prompt makes the model switch goals about 94% of the time.
- Good news for monitoring: in the open-weight models tested, goals stay anchored to readable context, which favors monitoring prompts and chains of thought and describing model goals directly in language.
Not Yet Confirmed
- The study covers open-weight models up to 106B parameters; whether the conclusions hold for larger closed models (with RLHF and other post-training) remains unclear.
Why It Matters
- As agents are increasingly let loose on long-horizon tasks, this work turns "what does the AI want?" from a philosophical question into an experimentally testable one, and provides a technical path for reading and monitoring model goals via natural language.
2026-11-20 ~ 2026-11-20 · 7 related posts
Primary sources
- Before training goal-pursuing agents, this new paper defines what an AI goal actually is inside LLMs — davidbau ·
- Goal Encoding in LLMs: sparse attention heads reveal and transplant model goals before execution — davidbau ·
- Swap the cached KV, swap the goal: LLMs recompute goals each step, study finds ~94% switching rate — davidbau ·
- Do language models have explicit goal encodings? New interpretability paper takes first step — benno_krojer · 2026-11-20
- Training agentic AI means training goals — but what is an AI goal, really? — davidbau · 2026-11-20
- [source] Before training goal-pursuing agents, this new paper defines what an AI goal actually is inside LLMs — davidbau · 2026-11-20
- LLM goals can be verbalized in English before execution and don't persist across steps, study shows — davidbau · 2026-11-20
- [source] Swap the cached KV, swap the goal: LLMs recompute goals each step, study finds ~94% switching rate — davidbau · 2026-11-20
- [source] Goal Encoding in LLMs: sparse attention heads reveal and transplant model goals before execution — davidbau · 2026-11-20
- Study: LLM goals stay anchored in readable context — up to 106B params — davidbau · 2026-11-20