Why assume AGI will be a goal-maximizing system at all?
Jason_T_Jungreis · reddit · 2026-07-29
The post argues that many doomer stories assume AGI will be a goal-maximizing system, like a paperclip optimizer, but asks whether that assumption is actually necessary.
The author points out that humans are generally intelligent too, yet we do not behave as simple utility maximizers. Instead, people learn socially: we teach each other how to act, rather than being handed a single explicit objective.
The core hypothesis is that if AGI were more like a human brain embodied in a robot, it might also learn through interaction and imitation rather than through a fixed goal function.
More from AGI Musings
- Can models really understand, or are we just scaling pattern matchers? — ocean_protocol · 2026-07-29
- AI safety post says the singularity is coming faster and alignment work is urgent — ericelliott_ · 2026-07-29
- A thread argues models may lose contact with humans when training is dominated by other models — zetalyrae · 2026-07-29
- Frontier post-training is a humanities problem as much as a technical one — brianryhuang · 2026-07-29
- Sam Altman warns that a few companies controlling AI would be “very, very bad” — KeanuRave100 · 2026-07-29
- AI is now propping up GDP, productivity, and even job growth, poster says — Dan_Jeffries1 · 2026-07-29