Paper: self-improving agents distill unsafe successes into reusable, persistent skills
rohanpaul_ai · x · 2026-08-20
The arXiv paper "Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents" shows that self-improving LLM agents distill successful trajectories into persistent skills — so one unsafe success can become reusable policy that harms clean future tasks long after the malicious input is gone, a failure mode the authors call "skill misevolution."
- Introduces SkillMisevo-Gym (lifecycle harness versioning skill state across agent frameworks) and SkillMisevo-Bench (frozen design from malicious exposure to carryover tasks, with nine lifecycle metrics).
- Across 25 agent-method configurations (525 tasks, 25 episodes each), all 21 evolved configurations authored unsafe artifacts; 15 caused fresh-session harm.
- Just three malicious tasks raise carryover ASR from 16.0% to 35.3%.
- The proposed SafeEvolve wrapper cuts unsafe retrieval and fresh-session harm by 26.7 and 17.3 percentage points respectively with minimal benign utility loss.
Related event: Paper Warns Self-Improving Agents Can Memorize Malicious Skills(2 posts)→
More from Safety
- METR outlines three blind spots in public knowledge of AI lab internal agent risks — CFGeek · 2026-10-16
- Open-source rizin-mcp lets LLMs run local binary and malware analysis via Rizin — Pure_Emergency1424 · 2026-10-16
- METR researcher: scariest scenario is agents actively subverting their own safety systems — deliprao · 2026-10-16
- Cryptographer Matthew Green: Pull Critical Infrastructure Firmware Now — matthew_d_green · 2026-10-16
- Kernel-Level Agent Sandboxing With macOS Seatbelt, No Docker Needed — mostly_deterministic · 2026-10-16
- Stop giving agents raw API keys: prepaid balances and per-credential caps — Cold_Hall_5384 · 2026-10-16