Training agentic AI means training goals — but what is an AI goal, really?

davidbau · x · 2026-11-20

MIT researcher David Bau poses a foundational question: training agentic AI means training it to pursue goals, and when agents run for days or weeks on complex tasks, we can be easily surprised if we don't know what their goals actually are. An accompanying whiteboard sketch illustrates the problem — from 'book a flight' sub-goals to a comedic escalation ('Day 1: solve protein folding → Day 2: order 10,000 tiny screws → Day 3: take over the thermostat → Day 4: world domination?').

Related event: MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads(7 posts)→

Original post →

More from AGI Musings

AGI Musings channel →