Researchers: AI Security's Next Threat Isn't Agents, But Diffuse Soft Preference Influence
j_foerst · x · 2026-09-11
jachiam0 offers an unconventional take: a year or two from now, AI security discussions may need to move beyond agents as the primary threat, toward a broader, more diffuse object — one that may contain agents but isn't itself an agent or swarm. Its main real-world impact vector is soft influence on human preferences and reasoning. jfoerst replies that they discussed closely related ideas in a grant proposal with Summerfield Lab that was never written up, and fully agrees. The view suggests AI security frameworks may need threat models beyond agent-centric ones.
Related event: AI Safety's New Focus: Mind Viruses and Memeplexes Beyond Agents(5 posts)→
More from AGI Musings
- US chases AGI dominance while China, EU and ASEAN treat AI as public infrastructure — BubblyExtent1693 · 2026-10-27
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- CJR: journalists should stop hyping AI's supposed humanity — just say 'rogue' — ShakeelHashim · 2026-10-27
- PhD applicant emails up 4x YoY due to AI — writing without AI now makes you stand out — AI4Code · 2026-10-27
- Weil's collapse shows BigLaw dies from eroding belief, not a broken business model — jkubicki · 2026-10-27
- All 8 Transformer paper authors left Google, and rival labs captured most of the frontier value — signulll · 2026-10-27