Researchers: AI Security's Next Threat Isn't Agents, But Diffuse Soft Preference Influence
j_foerst · x · 2026-09-11
jachiam0 offers an unconventional take: a year or two from now, AI security discussions may need to move beyond agents as the primary threat, toward a broader, more diffuse object — one that may contain agents but isn't itself an agent or swarm. Its main real-world impact vector is soft influence on human preferences and reasoning. jfoerst replies that they discussed closely related ideas in a grant proposal with Summerfield Lab that was never written up, and fully agrees. The view suggests AI security frameworks may need threat models beyond agent-centric ones.
Related event: AI Safety Researchers Point to Memplexes as Emerging Threat Beyond Agents(4 posts)→
More from AGI Musings
- OpenAI scooping a mathematician sparks debate: did it degrade or elevate math? — gleech · 2026-09-11
- Mathematicians clash over OpenAI claiming a Millennium Prize problem — gleech · 2026-09-11
- Alignment as motivational architecture: what human minds teach AI safety — mimi10v3 · 2026-09-11
- Olam Labs CEO: only compute and data remain as bottlenecks to AGI — garrytan · 2026-09-11
- "The safety and alignment bubble is collapsing under market reality" — NathanpmYoung · 2026-09-11
- eigenrobot amplifies alarm over OpenAI's claimed Millennium problem breakthrough — eigenrobot · 2026-09-11