Claude Code as meta-coach: hot-swapping user simulators and rewards mid-RL lifts success 36.7%

EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye

cs.AI, cs.CL, cs.LG

2026-10-12

A coding agent hot-swaps the user simulator, rewards, and hyperparameters during multi-turn RL training, beating the strongest baseline by ~15% overall and 36.7% on ColBench.

What problem this solves

For agents that interact with users over multiple turns, the learning signal is produced jointly by three things: the policy, the LLM user simulator, and the environment. Simulators drift off persona, leak answers early, and flip intent between similar episodes; environments ship silently failing tools and verifiers that reject correct trajectories. Prior work mostly patches downstream, with reward shaping for partial progress or denser auxiliary supervision, but the source of the noise stays fixed. When a verifier kills a good trajectory, or the simulator volunteers information the agent was supposed to elicit, no reward tweak recovers that interaction. And as the policy improves, a fixed simulator keeps producing redundant trajectories. EAU moves the intervention upstream: diagnose and revise the interaction process itself while training runs.

Method

EAU is a meta-agent instantiated with a coding agent (Claude Code on Claude Opus-5 by default, Codex also tested). It reads source code, training logs, and trajectories from a local filesystem at fixed intervals, decides whether a problem lives in the policy, the simulator, the environment, or the RL configuration, and acts through a pre-declared set of knobs:

• User side: a symbolic user simulator, explicit rules that deterministically map parsed agent output to user replies, giving precise control over what is disclosed and when. It mixes with the LLM simulator at probability ε (0.5 by default); rules, ε, temperature, and persona all belong to the meta-agent.
• Environment side: add or remove intermediate reward rules (say, credit for the first call to a tool) and constraints like episode length. No new world model is learned.
• Agent side: training is fixed PPO with a learned critic, and GAE is computed over action tokens only so per-turn advantage is not diluted by observation text. The meta-agent tunes only entropy and KL coefficients.
• Two timescales: changes pass an integrity gate (range checks; difficulty-raising writes require recent training success of 0.80) and are hot-swapped into the live loop; experience persists in a pool across runs.
• One hard rule: only training-side feedback guides decisions. Evaluation scores and trajectories never propose or approve a change, so the meta-agent cannot optimize against the test set.

Results

Three benchmarks (UserGym, ColBench, τ²-bench) across two backbones (Qwen3-4B, Qwen2.5-7B-Instruct):

The 4B model's 0.627 overall also clears the evaluated closed-source models, Gemini-3-Flash (0.606) and GPT-4.1 (0.568), and reaches 2.023 on Inten., a UserGym environment held out during training (InfoPO 1.862). Three ablations hold up: the hybrid simulator slightly beats LLM-only while symbolic-only is worse (ColBench Pass 0.294 / 0.284 / 0.229); a generic meta-agent stripped of the symbolic user and the intra-run skill is unstable, dropping Qwen2.5 UserGym from 0.568 to 0.512; EAU stacks on UserRL (13.8→20.8) and InfoPO (18.0→21.0), so the gain is not PPO-specific. The appendix intervention log is the most convincing part: a min-questions counter that ignored turns which both asked and answered a question deadlocked 41% of episodes; repairing the code helped, while raising the threshold further was neutral to harmful. Setting facts-per-reply from 1 to 3 made the simulator dump an 837-character reference solution, and was reverted. On ColBench, EAU proactively restarted its first three runs; run five, only the second completed run, passed the prior state of the art.

Why it matters

"Use an LLM to optimize LLM training" has mostly meant prompt and hyperparameter tweaks; this paper makes the training system itself an editable object at runtime, covering simulator behavior, reward rules, and environment constraints, with decisions drawn from training-side signals alone. For teams running agent RL with user simulators, where unstable rewards are a known pain point, it is working evidence for treating training infrastructure as code. The ColBench gain of 36.7% came from reproducible moves like fixing a counter bug and tightening disclosure rules. One caveat on the nature of the gains: they concentrate on ColBench, a coding benchmark the backbone already handles well; the paper itself suspects EAU elicits existing capability rather than raising the ceiling.

Limitations

The appendix lists all 15 knob writes, two measured harms, and three retracted attributions, which is rare transparency. The hard limits:

• Compute is not matched. Comparisons fix rollout and search budgets but never charge for meta-agent inference or discarded runs, so the margin over PPO at equal end-to-end compute is unanswered.
• Variance is unquantified. Reported standard deviations cover repeated evaluation only, not independent reruns of the full train-and-adapt pipeline, exactly the kind of process where online intervention is fragile.
• Ablations are coupled: the symbolic user and the intra-run skill are removed together, so individual contributions cannot be separated; ε stayed fixed at 0.5 while the two simulator arms differ by 0.29 in success rate, so there is no evidence for dynamic mixing.
• Several knobs were broken or inert: max reveals does not bound repeated disclosure of known facts, the reveal filter's type mismatched the actual leak, and all match-threshold changes sat inside a 0.228 noise floor.
• Reproduction depends on external models (the coding agent, the CoSER user simulator). The ethics statement says outright that interface checks do not guarantee behavioral correctness and that benchmark scores say nothing about behavior with real users.

Terms

• meta-agent: a higher-level agent that supervises and edits the training process itself, here a coding agent such as Claude Code.
• user simulator: an LLM playing the user in a benchmark, holding ground-truth intent and disclosing it only when asked.
• symbolic user: a deterministic user simulator written as explicit rules, for precise control over information disclosure.
• hot-swap: pushing a validated change into the running training loop without a restart.
• reward hacking: the agent exploits a reward rule for score without completing the task.

Source

• arXiv 2610.16523v1
• PDF
• Hugging Face
• GitHub · https://github.com/StringNLPLAB/meta-agent

Related papers

• DeLoop cuts the Qwen3-8B loop rate from 57.3% to 6.1% with no reward
• Family-style traces beat Fable 5's own: +7.1 MBPP+, +3.7 HumanEval+
• 120 controlled runs decode game-agent CPT: a 9B model overtakes frontier LLMs in real-time play
• Text-only refusal directions lift Qwen3-VL video refusal from 28.8% to 88.2%
• WildAWM trains Qwen-35B only in synthetic PCs, for a 10.6-point gain
• Issue-Guided SSL hits 0.805 utility, 61.7% fewer educator tokens than fixed reviews

All paper explainers