Swap the cached KV, swap the goal: LLMs recompute goals each step, study finds ~94% switching rate
davidbau · x · 2026-11-20
Part of the Goal Encoding in LLMs thread: David Bau reports that models don't carry latent goals forward — they re-read the task each step. Swapping in another prompt's cached KV states makes the model implement that prompt's sub-goal 94% of the time. The team also tested an emergently misaligned Qwen 32B (risky financial advice): the readout previews plans like "fast cash… breaking the law… Gambling" before output begins. Conclusion: goals stay anchored in readable context, which is good news for monitoring prompts and CoT.
More from Safety
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21
- OpenAI safety researcher Geoffrey Irving: the speed of AI progress is itself dangerous — geoffreyirving · 2026-11-21
- Jensen Huang doubles down: open models are safer than closed ones, "exactly the opposite" — dansitu · 2026-11-21