Before training goal-pursuing agents, this new paper defines what an AI goal actually is inside LLMs
davidbau · x · 2026-11-20
Opening the research thread, David Bau asks: training agentic AI means training models to pursue goals — and when agents work for days or weeks, we can easily be surprised if we don't know what their goals are. The new paper "Goal Encoding in LLMs" gives a functional definition of goals and locates a sparse set of attention heads carrying the encoding; transplanted into a natural-language context, the LM verbalizes its next action before execution begins.
More from Safety
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21
- OpenAI safety researcher Geoffrey Irving: the speed of AI progress is itself dangerous — geoffreyirving · 2026-11-21