ACE-RL Trains Agents to Curate Their Own Playbooks via Experience Handoff

sherryyangML · x · 2026-11-21

Problem: Scalar-reward agentic RL discards fine-grained environment feedback such as error logs. Recent harnesses like ACE let agents write notes/playbooks for future rollouts, but the construction of those artifacts is never learned.

Method (ACE-RL): Fixing the Agentic Context Engineering harness, one model is trained to act, reflect, and curate task-local playbooks, with a separate RL objective per role. Candidate experience-handoff artifacts are scored by their downstream effect on future actor rollouts, co-optimizing the action generator, insight reflector, and playbook curator.

Results: ACE-RL beats baselines on held-out agentic benchmarks, suggesting agent training should optimize not just current-rollout actions but the model-written experience future rollouts inherit.

Paper arXiv:2611.16409 by Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang; code released.

Original post →

More from coding & agent

coding & agent channel →