AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale
Minbyul Jeong, Chanwoong Yoon
cs.CL, cs.AI
2026-08-21
AgentMercury synthesizes 4,783 business worlds, then samples tasks. Qwen3.5-4B rises 12.3→15.7 on EnterpriseOps-Gym and 45.9→56.0 on AIME26; world authoring jumps 3.3%→83.3%.
Most agent environments are built around a task. You pick a problem, then you assemble a simulator that can run that problem. More tasks do not automatically mean more worlds. Real workflows run the other way. Tickets, mail, CRM records, and inventory already exist; tasks bubble out of a living world.
AgentMercury promotes world construction to a first-class role named Planet. A high-level business scenario is compiled into an executable world first. Tasks are sampled from that world later. The training worlds are not designed around the evaluation benches.
Planet maps a scenario σ to a world w: company identity, service graph, state schema, initial state, and world-level invariants. Invariants are not baked into the transition function. They become executable checks, usually SQL, run after the episode to catch cross-service constraints. If an upstream event must create a downstream record, the environment will not write that record for the agent. The same invariants have two views: a visible policy document the agent has to find, and a hidden verifier used for grading.
One world can be seeded into many tasks, each with its own instruction and rubric. The agent sees tool outputs, not the full state. Transitions are executed by the environment, so trajectories can be replayed from the action log and scored deterministically.
The release contains 4,783 executable environments across 14 industries and 50 countries. RL used 4,326 of those companies and 43,300 tasks, ten seeds per world. There are 842 unique tools, 16.1 tools per world on average, 2.9 services and 5.4 assertions per task; 25,991 tasks carry cross-system action risk. Only 3,200 tasks, about 7.4%, received gradients, yet they still covered 2,313 environments and 638 tools.
The 4B policy is trained with GRPO; the 35B MoE model is also trained with single-rollout SAO. Evaluation uses EnterpriseOps-Gym plus AIME26, HMMT, LiveCodeBench, SciCode, τ³, BFCL, and GPQA-Diamond, three runs each.
World authoring is trained as well. Qwen3.5-35B-A3B is fine-tuned on 29,823 construction traces covering brief-to-world generation, mid-stage completion, validator-guided repair, and intent-to-diff. Success on 30 held-out briefs requires passing all 12 structural validators.
Qwen3.5-4B with GRPO moves EnterpriseOps-Gym from 12.3 to 15.7, a 27.6% relative gain. Drive goes from 6.2 to 15.6, Email from 23.9 to 33.3. CSM falls from 9.2 to 5.6, the only drop among eight domains. The 35B model goes from 24.8 to 28.1 with GRPO and 28.3 with SAO, up in every domain. GPT-5 sits at 30.9 on the same bench; the trained 35B still trails it.
Transfer off the training distribution is the sharper result, because none of these benches entered world construction:
| Bench | 4B base | 4B+GRPO |
| AIME26 | 45.9 | 56.0 |
| HMMT | 28.5 | 35.4 |
| LiveCodeBench | 36.6 | 44.0 |
| SciCode | 22.6 | 25.7 |
| BFCL | 30.3 | 31.7 |
At 35B, math is near the ceiling, AIME26 91.0 to 92.2, while tool-facing BFCL jumps from 31.1 to 42.1. On τ³ the base 35B is extremely noisy (Telecom 49.1±49.8); SAO lifts the mean and still leaves a wide band. Training reward climbs, truncation falls, and degenerate replies stay near zero.
World authoring: the 35B base passes 3.3% of briefs zero-shot and 83.3% after fine-tuning, matching Claude Opus 4.8 and DeepSeek-V4-Pro. A construction recipe lifts the base from 3.3% to 20.0% and wrecks the fine-tuned model, 83.3% down to 10.0%, with 27 of 30 briefs failing the cross-service check. API models land roughly between 66.7% and 83.3% zero-shot; the recipe does not help them consistently. GPT-5.4 collapses the trigger and target of an invariant into one service on 10 of 30 briefs.
The claim is concrete: scale the structure of worlds, not the count of benchmark clones. Business workflows already contain long-horizon state, mixed tools, and cross-service constraints, and those interaction patterns move into math, code, and tool use. Building the world can itself be learned.
Keep the enterprise number in perspective. 15.7 for a 4B on EnterpriseOps, against GPT-5 at 30.9, is not a deployable office agent. The 10-point AIME26 jump is the result that needs an explanation, because the training worlds never saw contest math.
The loop is still open. The generator does not mint new scenarios from policy failures, and no learned world model previews a candidate world. SAO was unstable at 4B and was dropped, so the environment signal is not algorithm-agnostic on small models. The 3.6-point CSM drop is unexplained. RL touched 7.4% of tasks. Authoring is measured on 30 briefs. The software is synthetic, not Salesforce or Jira. Invariants are checked with post-hoc SQL; whether the visible policy and the hidden verifier actually match is not human-rated.