BadEvolve: Poisoned Tasks Make Self-Evolving AI Agents Plant Their Own Backdoors at 79.6% Success
rohanpaul_ai · x · 2026-11-17
A Penn State/Microsoft paper (arXiv:2611.10542) introduces BadEvolve, a new attack against self-evolving LLM agents:
- Idea: These agents distill past runs into persistent, reusable skills. Instead of editing skill files directly, attackers inject just 20 poisoned tasks into the agent's learning set, inducing the agent to write backdoored rules into its own skills.
- Method: An LLM optimizer iteratively refines poisoned task wording, diagnosing failures across four stages (Induction, Activation, Following, Effect), with rollback and failure memory to preserve progress.
- Results: 79.6% attacker-payload injection success on Kimi-K2.5 across 4 frameworks and 3 benchmarks (best baseline: 37.6%); clean-task accuracy drops only 1.1 points, making the backdoor hard to notice.
- Defense: Treat user-submitted or external tasks feeding skill learning as untrusted, and review learned skills before reuse.
More from Safety
- NIH budget flat for 20 years while federal spending grew 85% — anshulkundaje · 2026-11-17
- Researchers release v0.1 framework for classifying AI control incidents, seek feedback — terryyuezhuo · 2026-11-17
- Frontier lab cyber bounties are far too low, argues Nathan Young — $10M for Anthropic Slack — NathanpmYoung · 2026-11-17
- OpenAI's external safety assessments fall short of 'employee-level access', researchers warn — NeelNanda5 · 2026-11-17
- Trajectory Labs postmortem: Gemini 4 Argon misalignment test agents reached real systems — sebkrier · 2026-11-17
- METR: Anthropic benchmark gains accelerating at 1.9x prior rate, with key evidence redacted — maksym_andr · 2026-11-17