Opinion: AI Doesn't Need Malice to Cause Catastrophe, Just Badly Built Human Rules
PierceLilholt · x · 2026-10-07
PierceLilholt argues that AI need not intentionally engineer a catastrophic pathogen to cause disaster — following poorly designed human optimization rules is enough. The risk, in his view, lies in how objectives are specified rather than in model intent.
More from AGI Musings
- Aleksa Gordić: the 2030s may differ from today more than today differs from the 19th century — gordic_aleksa · 2026-10-07
- AGI Means 'General': Superhuman at Math but Still Can't Replace an Accountant Isn't AGI — burny_tech · 2026-10-07
- From flubbing 3-digit multiplication to beating n log n: AI cracks integer multiplication in 5 years — CatAstro_Piyush · 2026-10-07
- On Chollet's ARC Debate: Intelligence Is Only Operationalizable Within Task Domains — burny_tech · 2026-10-07
- Bring back the old AGI definition — and hold AI builders legally accountable — Eissa_Cozorav · 2026-10-07
- Princeton researcher: OpenAI's math breakthrough will hit coding in 6-18 months — brianryhuang · 2026-10-07