LLM goals can be verbalized in English before execution and don't persist across steps, study shows
davidbau · x · 2026-11-20
David Bau summarizes two surprises from Arnab Sen Sharma's goal research: (1) a sparse set of attention heads is causal for the model's next goal, and patching the signal into a verbalizer prompt makes the LM describe its upcoming action in English before execution; (2) models don't carry latent goals forward — each step re-reads the task and recomputes, so swapping another prompt's cached KV states makes it implement that prompt's sub-goal 94% of the time. Goals staying anchored in readable context is good news for monitoring prompts, CoT, and verbalizing goals directly.
More from Safety
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21
- OpenAI safety researcher Geoffrey Irving: the speed of AI progress is itself dangerous — geoffreyirving · 2026-11-21