Study: coding-agent self-evolution works in only 2 of 7 tested /retro changes
dotey · x · 2026-11-04
A paper by Christoph Treude (SMU) and Sebastian Baltes (Heidelberg) tested whether letting Claude Code run /retro after tasks and amend its harness actually helps.
- 30 SWE-bench Verified issues generated 125 retrospective suggestions; 24 were implemented, 11 testable
- Of 7 evaluated sessions, only 2 changes fully eliminated the targeted problem; some added unnecessary checks on unrelated tasks
- 18/43 proposals were generic advice unrelated to what actually happened
- Conclusion: harness self-evolution needs human selection, evaluation, versioning, and rollback
dotey adds: only skills with programmatically verifiable acceptance criteria can self-improve; humans should direct, AI should edit, with version control per iteration.
Related event: Paper Evaluates Self-Evolving Claude Code Skill /retro(3 posts)→
More from coding & agent
- Inkling-2-Mini hits 10.04x on KernelBench-Mega locally, beating Grok 4.7 and GPT-6.1 — ziqiao_ma · 2026-11-23
- Visa launches CLI agent wallet: verify AI agents with a Visa card to earn OUSD — Scobleizer · 2026-11-23
- Why TLA+ users should read Lamport's original TLA paper, in the age of AI-written specs — tianyin_xu · 2026-11-23
- Ditching Mermaid: flight-board style table layouts for agent workflow UIs — thesephist · 2026-11-23
- Turn Claude Code + a free Obsidian vault into a second brain in 5 minutes — dr_cintas · 2026-11-23
- LinkedIn cuts data export in December: contacts' emails and titles dropped — thedealdirector · 2026-11-23