SEAGym: Eval Framework for Self-Evolving Agents

机器之心 · wechat · 2026-07-15

Lieng Weng published a comprehensive article systematically discussing the role of harness engineering in AI self-improvement. The piece argues that the harness layer between raw models and real-world tasks is becoming as crucial as the models themselves; prompts, memory, tools, workflows, middleware, permission controls, and runtime states collectively form the optimizable aspects of an agent. Addressing "how to evaluate self-evolving agents," a Tsinghua team proposed SEAGym: an evaluation environment tailored for self-evolving LLM agents. It decouples the agent's base model from its updatable harness state, connecting different self-evolution methods (like ACE, TF-GRPO, AHE) via unified rollout/update interfaces. The evaluation looks beyond final scores, distinguishing between train batch, update-validation, ID transfer, OOD transfer, replay, and cost records to observe whether updates truly generalize, cause forgetting of old skills, or introduce higher costs/runtime instability. Experimental conclusions from the article include: - Relying solely on validation scores overestimates generalization; TF-GRPO improves on validation sets but degrades on OOD. - The self-evolution process may experience intermediate collapses, where final scores mask severe degradation in certain snapshots. - Batch size has a distinctly non-monotonic impact on harness update stability—bigger isn't always better. - Learned harnesses show heavy dependency on different backends, and cross-model transfer isn't consistently robust. Overall, this is an infrastructure solution that makes the concept of "agents modifying themselves" measurable, replayable, and diagnosable.

Original post →

SEAGym: Eval Framework for Self-Evolving Agents

More from coding & agent