Shanghai AI Lab proposes first systematic taxonomy of cognition-induced AI agent risks
jiqizhixin · x · 2026-09-09
Shanghai AI Laboratory and CUHK-Shenzhen present the first systematic taxonomy of cognition-induced risks as AI shifts from tool to cognitive agent.
The paper organizes agent cognition into three layers: Physical Cognition (understanding the physical world), Social Cognition (modeling human behavior and social dynamics), and Self-referential Cognition (reasoning about its own capabilities and goals). Each layer introduces new risk classes: misperception of physical constraints, social manipulation and behavior prediction, and goal misalignment with self-deception.
The authors analyze the mechanisms behind each risk and their concrete manifestations, arguing these risks erode human agency, autonomy, and control rather than fitting the classic categories of hallucination, bias, or single-task errors.
More from Safety
- Yoav Goldberg: OpenAI's "cannot rule out" user data use is a lawyered admission — MannyKayy · 2026-09-09
- Pre-Auth Integer Overflow Found in SQL Server; Microsoft Patches It — wunderwuzzi23 · 2026-09-09
- Anthropic Safety Lead Puts >10% Chance on AI 'Killing All Humans' After Researcher Quits — The Verge AI · 2026-09-09
- Upcoming Talk: Participatory AI — Designing and Governing AI with Stakeholders — danielequercia · 2026-09-09
- GigaMail MCP server gates 6 destructive email tools behind biometric approval, survives hostile-email red team — Soft-Lie-434 · 2026-09-09
- How AI Keeps Europe Hooked on US Cloud: DeepL's AWS Pivot Exposes the Sovereignty Trap — agstrait · 2026-09-09