Why the AI safety and security camps think so differently: mystics vs engineers
sebkrier · x · 2026-10-22
Nat Purser shares an observation from her early days in AI policy: ML researchers don't act like typical techies — they're "mystics," with a self-styled-prophet online presence.
- The cyber security camp treats systems as understandable, assuming risks can be mitigated through standard engineering and "defense in depth."
- The ML/alignment camp emphasizes — and sometimes delights in — the unknowability of "grown, not engineered" AI systems, focusing on emergent behaviors their creators don't fully understand.
She argues these philosophies flow naturally from the kinds of systems each group has experience with, explaining a persistent rift between two camps that should be natural allies.
More from AGI Musings
- 'Almost right' AI at scale is the real risk, argues assurance infrastructure builder — AryHHAry · 2026-10-22
- Nadella touts Microsoft's Nature paper on ML forecasts for the 2-6 week window — LesterMackey · 2026-10-22
- Microsoft, MIT and ECMWF's Nature paper doubles 2-6 week forecast accuracy with open-source CPU bias correction — LesterMackey · 2026-10-22
- Executive advice: visit a country that fully blocks your AI to feel the coming dependency shock — perilli · 2026-10-22
- Data is not software: company ground truth lives in heads, not a global repo — HanchungLee · 2026-10-22
- Blogger predicts Apple's craftporn aesthetic will make people miss the AI slop era — yungcontent · 2026-10-22