PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, Ling Yang
cs.CL
2026-08-05
PAST-Bench tests whether retained experience makes personal agents better: 26 scenarios, 204 episodes, seven base models. Gains are real but uneven, and the same headline score can hide very different mechanisms.
Personal agents persist across sessions, keeping your preferences, task histories, tool routines, and learned skills. The intuition is that retained experience should make them better the more you use them. But whether that actually happens has never been tested cleanly. An agent that does well on a second task might be benefiting from memory, or it might just be a strong base model, a smooth prompt, an easy task, or scoring noise. Existing benchmarks measure one-shot task performance and conflate base capability with genuine self-improvement.
PAST-Bench isolates this attribution problem. It defines "online self-evolution": an agent reuses past experience to change future behavior without retraining, prompt tuning, or long-context stuffing. The benchmark delivers an attribution answer, not a leaderboard.
PAST-Bench spans 26 scenarios and 204 episodes across four capabilities: Memory (declarative facts like preferences and policies, 5 families / 41 episodes), Procedural Reuse (technical SOPs and deploy pipelines, 8 / 64), Information Gathering (proactive retrieval under noise, 6 / 48), and Update (new authoritative values overriding old ones, 7 / 51).
Each task family runs four ordered roles: Cold (no persistence yet), Learn (write target info), Evaluation (probe reuse with trigger wording removed), and Control (confirm gains are not from shortcuts). The core device is a matched ablation: same prompt, grader, tool stack, and seed, run twice with persistence off and on. Two metrics matter: the self-evolution gap (on minus off) and the mechanism-evidence score, which checks whether a gain actually followed the intended write, retrieve, and correctly-apply pathway.
Across seven base models (GLM-5.1, Kimi K2.6, DeepSeek-V4-Pro, MiniMax-M2.7, GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.6), persistence always helps, with overall gain from +0.13 to +0.24. GPT-5.4 is strongest (+0.24, mechanism score 0.80). Gains are uneven across capabilities, and the key finding is that two agents with the same headline gain can have very different mechanism scores, meaning the same number can hide very different mechanisms.
The authors then build Hermes+, adding five runtime mechanisms (plan-time consultation, typed memory bindings, ranked patchable skills, a retrieval gate that blocks draft answers when no memory read occurred, and synchronous closeout). It lifts the Update gain from +0.12 to +0.24 and transfers across five base models. The honest number: overall gain is 0.13 plus or minus 0.04 for Hermes versus 0.15 plus or minus 0.06 for Hermes+, so the +0.02 overall gain is smaller than run-to-run noise. The clean improvement is on Update only.
Anyone building long-term memory, personalization, or agent frameworks should read this. It is a measurement-science warning: a reported "agents get better with use" can be base strength, noise, or luck. Only a mechanism score makes it credible. Framework builders can also map themselves against the five mechanisms, most of which lack a retrieval gate and synchronous closeout.
The authors list several: task families are synthetic and evaluated in isolation; only current sequences are tested, not cross-family transfer; the four capabilities are foundations, not the full space of recursive improvement; the mechanism score measures pathway consistency, not causal necessity, which would need counterfactual interventions. One more caveat: the score assumes a single semantically valid persistence pathway, but different agents may encode the same experience as memory, skill, structured doc, or revised policy, all reasonable.