Before training goal-pursuing agents, this new paper defines what an AI goal actually is inside LLMs

davidbau · x · 2026-11-20

Opening the research thread, David Bau asks: training agentic AI means training models to pursue goals — and when agents work for days or weeks, we can easily be surprised if we don't know what their goals are. The new paper "Goal Encoding in LLMs" gives a functional definition of goals and locates a sparse set of attention heads carrying the encoding; transplanted into a natural-language context, the LM verbalizes its next action before execution begins.

Related event: MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads(7 posts)→

Original post →

More from Safety

Safety channel →