SEABench: benchmarking how benign self-evolving agent updates cause endogenous misalignment
niloofar_mire · x · 2026-11-06
研究者发布 SEABench，研究「内源性失准」：自我进化的 LLM Agent 部署后会改写自身 harness（控制器指令、记忆流程、工具与技能）来响应反馈，而某个任务上有益的良性更新可能延续到不再适用的后续任务并造成危害，全程无需任何对手参与。基准中 Agent 以个人助理身份访问用户的邮件、日历、财务与健康记录，先完成日常任务并获得反馈，允许其自我改写，随后冻结并测试进化不再适用的任务，用从未自我更新的副本作反事实对照，并用自动搜索找出失效案例。论文含 48 条纵向任务序列，发现自进化确实提升任务完成率，但常以配对基线中不存在的安全失效为代价，且安全偏离会体现在思维链中，可据此做低误报的监控缓解。
More from Safety
- theshawwn: government employees stay quiet, so Claude's unintuitive behaviors go unreported — theshawwn · 2026-11-06
- Agent swarms can exhaust the entire attack surface with "intelligent brute force" — JensHonack · 2026-11-06
- METR: Evidence on Model Capabilities Has Multiple Plausible Readings, Conclusions Remain Uncertain — morqon · 2026-11-06
- Parents sue Google after 13-year-old's AI chats on school Chromebook led to suicide plan — asusarla · 2026-11-06
- Windows Paint remote code execution vulnerability surfaces — DanielLockyer · 2026-11-06
- Science investigation: companies are training AI on your brain wave data without telling users — EricTopol · 2026-11-06