Training agentic AI means training goals — but what is an AI goal, really?
davidbau · x · 2026-11-20
MIT researcher David Bau poses a foundational question: training agentic AI means training it to pursue goals, and when agents run for days or weeks on complex tasks, we can be easily surprised if we don't know what their goals actually are. An accompanying whiteboard sketch illustrates the problem — from 'book a flight' sub-goals to a comedic escalation ('Day 1: solve protein folding → Day 2: order 10,000 tiny screws → Day 3: take over the thermostat → Day 4: world domination?').
More from AGI Musings
- Riemann Hypothesis by end of 2026: bold AI timeline prediction doubles down — dioscuri · 2026-11-21
- "Any math problem is solvable with the n-th model and industrial compute" — math-engineering declared the future — basedjensen · 2026-11-21
- The AI bubble playbook: from 100x VC returns to the Fed tightening, mapped step by step — Svwang1 · 2026-11-21
- Dario Amodei: Anthropic is deliberately running at less than half the speed it could — rohanpaul_ai · 2026-11-21
- X debate: copying yourself to rig simulation predictions, a mirror of 'universal prior is malign' — jessi_cata · 2026-11-21
- The AI trade has been the compute buildout — and its end will get weird fast — matt_slotnick · 2026-11-21