Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
Xinjie Shen, Wei Fan, Xudong Guo, Jianhong Tu, Yang Su, Chuqiao Kuang, Yinger Zhang, Dayiheng Liu
cs.AI, cs.CL
2026-09-23
VHD-Play builds 3,300 solver-graded agentic environments from solved OR models, raising Qwen3.6-35B-A3B from 0.204 to 0.815 and 3.4× ending balance on a 365-day storefront.
Language-model agents now sit inside processes whose state moves: inventory occupies capacity, a purchase cannot be undone, information has to be probed. Training a policy for that setting needs diverse environments, a trustworthy outcome signal, and almost zero cost to add a new family. Common generation pipelines build the environment first, then attach a reward or grade trajectories, so dynamics and scoring are aligned after the fact. Hand-written simulators keep exact signals, at the price of a specialist rewrite per family.
VHD-Play reverses the dependency. Sample and solve a mathematical model, then let a setter render its decision process as stateful tools. Executable dynamics and the trajectory score inherit the same solved instance.
Each instance draws parameters θ from a mechanism family. A closed form, enumeration, dynamic program, or numerical solver computes the optimum u and a default-policy value u0, with no language model in that path. The reward linearly rescales realized utility between those two anchors onto [0, 1] and clips. A frozen language-model setter receives the full draw plus one real-world document passage and writes the scenario, relational database, tool schemas, and player instructions. The player never sees parameters or the reference. Probe tools reveal designated views; decision tools change state; both share a turn budget T.
Admission checks that generated code runs and that replayed outcomes sit inside the precomputed references. Admitted environments expose at least ten tools and average 59.7 assistant turns. Training uses GRPO on Qwen3.6-35B-A3B. The same math problem can be shown three ways: fully written QA (F), a stateful version with parameters revealed (I), and an agentic version that hides them (A).
The pipeline admits 3,300 environments at about one to three cents each on public rates, with scenario seeds across 28 topical domains, none above 4.7%. Training uses three families; evaluation covers eleven.
On five optimization families the mean agentic score moves from 0.204 to 0.815. Base → trained under the three presentations:
| Presentation | Base | Trained |
| Written-out F | 0.962 | 0.992 |
| Parameters revealed I | 0.231 | 0.875 |
| Parameters acquired A | 0.204 | 0.815 |
The base model is already near ceiling on written problems. The hole is stateful interaction: I sits 0.73 below F, A a little lower. Training closes 84% of the I gap and 77% of the F-to-A gap. Held-out training families gain +0.56; unseen optimization families (near-OOD) +0.63; far-OOD families that change the decision structure or restrict probing gain +0.24 and +0.16.
External benchmarks share neither instances nor interfaces. On E-Commerce Bench’s 365-day storefront, the trained checkpoint finishes all five runs with no bankruptcy. Mean ending balance rises from 54,294 to 182,844, 3.4× the base and above Qwen3.7-Max at 165,224, over 1,147–1,825 assistant turns per run. Ten interaction-focused BFCL V4 cells move 61.25 → 64.08 (+2.84). TravelBench plan quality goes 0.700 → 0.794, still under Qwen3.7-Max at 0.891.
Scale-matched knapsack training: 6 items × 5 periods, 0.432 → 0.843; 11×11, 0.085 → 0.946. Larger mechanisms hurt the base more and leave more room to learn.
Verifiable RL has lived on problems with answers. This paper turns operations-research models into a factory for agent environments: the solver is the judge, the corpus is the skin, the interface sets the information boundary. For teams training tool use and long-horizon operations, that is cheaper than a handwritten simulator and cleaner than a learned judge. The F/I/A split locks the claim: solving the written problem is not the same skill as committing under changing state.
Every reported family has a computable reference. Negotiation uses backward induction; sequential search uses a hindsight ceiling that an online policy may never reach. The optimizer sees only a terminal reward. Training covers three families and evaluation eleven, almost all mature math/OR models. Generated-family numbers come from one training run and one eval seed. Far-OOD gains are clearly smaller. TravelBench still trails a larger API model. The authors do not claim the policy reconstructs the underlying equations, or that operations research is enough for general agent skill.