Amazon paper: stale agent skills cut pass rate 11.4 points, selective maintenance fixes it
rohanpaul_ai · x · 2026-11-20
An Amazon/USC paper shows saved agent skills can silently go stale as environments change: on Terminal-Bench 2, a skill that had added 4.7 points for Claude Opus 4.7 left it 11.4 points below the no-skill baseline after a change; Haiku 4.5's IRS tax-form field accuracy dropped to 30.5% with a stale skill.
Their fix, selective maintenance, replays saved checks without extra model calls, repairs only conflicting skill content, and rebuilds when local repair is insufficient — lifting accuracy by up to 67pp over reusing stale state and cutting execution steps by up to 37%. Post-maintenance, Haiku's tax-form accuracy reached 97.5%.
Practical takeaway: validate saved agent skills against the current environment before reuse, instead of waiting for tasks to fail.
More from coding & agent
- Running a one-person company with hundreds of AI agents: 'Agent 1 takes the task you like least' — alvelda · 2026-11-20
- Replit's performance engineering: have the agent instrument, trace, optimize, and verify — amasad · 2026-11-20
- Mnemolith: an MCP memory server that flags stale memories when dependent code changes — Anish2910 · 2026-11-20
- Codex iOS update adds persistent resumable side chats, project picker and iPad shortcuts — SIGKITTEN · 2026-11-20
- The AI-era PRD template: prototypes show what, the PRD shows why and how to measure — aakashgupta · 2026-11-20
- Agent checks 40 startup names in one go for 3 cents via iMessage — socialwithaayan · 2026-11-20