The First Dangerous AI Won't Have Bad Intentions — It'll Be Great at Executing Ours
gixxerscott · reddit · 2026-09-11
The author argues the most plausible catastrophic AI risk isn't a malicious conscious machine, but an extremely capable one executing human intent.
- Historical barriers to mass harm — specialized teams, resources, organizations — are themselves protective; frontier models are systematically lowering them.
- With 8 billion humans, even a tiny malicious fraction matters; no collective irrationality required.
- Physical-world barriers remain but only need to get lower, not disappear.
- Poses the formula: Risk ≈ Capability × Autonomy × Access × Malicious Intent ÷ Physical Barriers.
- Implication: alignment alone isn't the whole problem — capability proliferation is, and it may be far harder to solve.
More from AGI Musings
- "Hard to find a group more sincere than the AI doom crowd," dev quips — jd_pressman · 2026-09-11
- Researcher urges parents to talk with kids about AI ethics and deskilling — LuizaJarovsky · 2026-09-11
- The p(doom) industry: turning 'a bad feeling' into a research program — YogeshMalik · 2026-09-11
- Anton Leicht: Send Independent Evaluators Into AI Labs Before the Next Incident Forces It — trevposts · 2026-09-11
- LinkedIn veteran recalls 2008: logistic regression plus better data beat all the PhD techniques — vboykis · 2026-09-11
- AGI Portfolio Update: 16.8% Return in 7 Months, Beating SPY by 4.4 Points — avilacjf · 2026-09-11