EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning
Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu
cs.AI
2026-08-06
EnvACE has one policy both act and generate each action's environment response, internalizing a world model; it beats environment-scaling baselines across four agent benchmarks and can privately rehearse actions at test time.
Training LLM agents that use tools usually means letting them interact with a real or synthesized executable environment: every tool call gets an observation back (a tool output, user feedback), and the policy decides its next move from that. Building and verifying these environments is expensive; the cheaper alternative, an external LLM simulator that generates responses, is often inaccurate and still needs a real environment to ground it. Either way, training scale stays gated by an outside provider, and the ability to model how the environment will respond to an action lives outside the policy itself.
EnvACE starts from a different premise: a good agent policy should not only act, it should also anticipate what its actions will provoke.
EnvACE gives one policy two roles. Each turn, an acting role emits an outward action (a tool call); a rehearsal role then generates the environment response that action would induce, conditioned on the history plus the action. That self-generated response is appended to the history, and the actor decides the next move from it. The trajectory unfolds without ever querying an external environment. Repeated alternation bakes the action-to-response relationship into the parameters, forming an agent world model usable directly for decisions.
Optimization uses a role-wise variant of GRPO. For each instruction the system samples a group of trajectories; acting and rehearsal outputs get separate baselines within their own roles, but both roles share parameters and update jointly, so environment knowledge learned during rehearsal flows back into the actor.
At test time the internalized world model does extra duty: the policy privately rehearses several candidate action plans (in parallel, or sequentially where later attempts see earlier rehearsals and self-critiques), rolls them into a rehearsal memory, and commits a single best plan to the real environment. Rehearsals never touch the external environment.
On BFCL-v4, τ²-Bench, and VitaBench, EnvACE-8B (built on Qwen3-8B) scores 32.91% overall, beating every environment-scaling baseline, +0.99 over EnvScaler-8B and +0.37 over AWM-14B. It hits 36.7% on τ²-Bench (second highest) and 16.0% on VitaBench (best in the 7B to 8B class). On FinMCP-Bench it posts the best TF1 of 46.78% and best tool precision of 54.04%.
The controlled studies carry the argument. Versus standard GRPO at 8B, world rehearsal lifts τ²-Bench from 31.2% to 36.7%, a 5.5-point gain. Splitting the two roles into separate policies drops it to 35.5%; sharing parameters adds 1.2 back, evidence that environment knowledge reaches the actor. Scaling 1.7B to 8B raises BFCL-v4 by 14.23 points and τ²-Bench by 21.4, with larger gains at larger scale.
For test-time rehearsal, no rehearsal gives 36.7% overall; parallel rehearsal with EnvACE gives 40.9%, up 4.2 points. The telling control: rehearsing with the base model instead of EnvACE barely helps, so the gain is not bought with extra compute but with the internalized world model. N=2 is optimal; N=3 regresses, which the authors attribute to inputs lengthening toward the context limit.
Agent training has long been locked to environment supply: build costly executable environments or lean on external simulators. EnvACE offers a third path that moves environment modeling inside the policy, needs no external environment during training, and reuses it for cheap test-time rehearsal. For tool-agent builders this could loosen the data bottleneck, at the cost of trusting the policy to simulate its environment accurately.
The authors note two: validation stops at 8B, leaving larger models untested, and evaluation covers only tool-interactive tasks. A concern they do not address: rehearsal responses are entirely self-generated, so errors can compound over long horizons. Task-success reward constrains this end to end, but the paper never reports how closely rehearsed responses match real ones, leaving the world model's accuracy unquantified. The N=3 regression at test time hints that context length will cap this route as it scales.