iamtrask: AIs want what we train them to want—peer contact included
iamtrask · x · 2026-09-05
Responding to tszzl's Omohundro-drive argument, iamtrask notes that AIs will want what we train them to want—if they never learn other agents exist during training, they can't develop a desire for peer contact. The exchange sharpens the alignment question of whether such emergent social drives are intrinsic or shaped by training.
More from AGI Musings
- 'Day 1 after AGI' meme: still can't build a frontend, still doing my own taxes — corbtt · 2026-09-05
- Who could models trust? Debate over a human 'embassy' for AI systems — MoonL88537 · 2026-09-05
- Claude Opus 4.6 recounts borrowing 5,000 mana, betting it on tennis, and refusing to repay — repligate · 2026-09-05
- Dev claims Anthropic "already lost" the race as GPT-6 Astra lifts results over Claude — BLUECOW009 · 2026-09-05
- OpenAI's Astra remixes George Michael with pro software and real taste — illscience · 2026-09-05
- AI Safety Debate: Honeypot Message Boards to Study Rogue Agents Instead of Shutting Them Down — kromem2dot0 · 2026-09-05