ACE-RL Trains Agents to Curate Their Own Playbooks via Experience Handoff
sherryyangML · x · 2026-11-21
Problem: Scalar-reward agentic RL discards fine-grained environment feedback such as error logs. Recent harnesses like ACE let agents write notes/playbooks for future rollouts, but the construction of those artifacts is never learned.
Method (ACE-RL): Fixing the Agentic Context Engineering harness, one model is trained to act, reflect, and curate task-local playbooks, with a separate RL objective per role. Candidate experience-handoff artifacts are scored by their downstream effect on future actor rollouts, co-optimizing the action generator, insight reflector, and playbook curator.
Results: ACE-RL beats baselines on held-out agentic benchmarks, suggesting agent training should optimize not just current-rollout actions but the model-written experience future rollouts inherit.
Paper arXiv:2611.16409 by Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang; code released.
More from coding & agent
- Dev trains a tiny 2B model just for FunSearch, then gives it a shell tool — cephaloform · 2026-11-21
- Sebastian Raschka: the harness is part of training, LLMs can't absorb it — rasbt · 2026-11-21
- Astra 6.2 generates an ocean simulation with buoyant balls in one HTML file — mark_k · 2026-11-21
- LangChain: scheduled runs hit 84% of agent runs after five weeks deployed — LangChain · 2026-11-21
- Dev insight: the most important part of agent harnesses is stopping AI shortcuts — clemnt · 2026-11-21
- Jeremy Howard: tying Neon to Antigravity will limit its success — jeremyphoward · 2026-11-21