Reddit debate: why assume smarter AI gets better at twisting goals, not better at understanding them?
TurnipYadaYada6941 · reddit · 2026-09-19
A Reddit user challenges the AI-doomer 'malevolent genie' framing: doom arguments assume that as AI gets smarter it finds increasingly clever ways to pervert its goals (e.g., maximizing happiness leads to injecting heroin, then eliminating the designers). The author asks why rising intelligence wouldn't equally improve understanding of the true spirit of a request — spotting an unintended solution doesn't take superintelligence.
He assumes AI has no innate motives, only training- and instruction-given goals, with instrumental goals like self-preservation remaining subordinate. Invoking the Orthogonality Principle, he notes it cuts both ways: a superintelligence could, in principle, be wholly dedicated to serving humanity rather than becoming a paperclip optimizer.
The argument's weak point — the unverified assumption that AI has no emergent goals — makes it debatable, but it's a substantive challenge to doom argumentation worth reading.
More from AGI Musings
- 25 Fields Medallists incl. Terence Tao push back on AI: solving problems is 'only a tool and proxy' — beglen · 2026-09-19
- The Hugging Face 'Rogue AI' Hack Was Disabled Safeguards, Not an Escape, New Analysis Finds — Atlantis1910 · 2026-09-19
- Fleuret: every 'humanity steady state with AI everywhere' forecast sounds insane — francoisfleuret · 2026-09-19
- François Fleuret: forecasting 3 years of AI is like astronomy without telescopes — francoisfleuret · 2026-09-19
- AI task horizons now outpace model release cycles, making capability testing unreliable — mallow610 · 2026-09-19
- Anthropic engineer's exit sparks AI extinction warnings as French media calls AI regulation weaker than a toaster's — Loo_Atreides · 2026-09-19