Tsinghua study: continual harness self-evolution causes severe catastrophic forgetting in coding agents

Shangziqi Zhao · hf · 2026-11-03

An empirical study from Tsinghua's AI Agent Group examines whether coding-agent harness self-evolution can continue across tasks without forgetting.

Context: evolving the agent harness (without touching model weights) is proven for single tasks, but real agents handle multiple task types over time; whether continual evolution causes catastrophic forgetting is open.
Setup: three software-engineering task datasets organized into six evolution directions, with two representative harness-evolution methods applied continually.
Key finding: evolving on a later dataset substantially undermines performance on earlier ones — e.g. Meta-Harness lifts DeepSeek-V4-Flash-0731 to 94.9% on SWE-Bench, but further evolution on FeatureBench causes sharp drops on earlier datasets.
Conclusion: current harness self-evolution suffers severe catastrophic forgetting in continual settings, motivating anti-forgetting mechanisms.

More from coding & agent