AutoGym: Blueprint-First Generation of Verifiable Agent Gyms
Aarati Andrea Noronha, Kavya Ravikumar, Carly Xiaoyu Lin
NeurIPS 2026 Workshop: Who Veri
cs.AI, cs.LG, cs.MA
2026-09-19
AutoGym generates verifiable agent gyms from a solution blueprint. Under C2, 39% of tasks are hard for Opus 4.6, versus 21.5% on a human suite and 4% on Agent World Model.
Training language models as agents with RL is now limited less by the learning algorithm than by the gym: a task spec, an executable environment, and a verifier that can tell success from failure. Hand-built suites such as WebArena, OSWorld, and SWE-bench are careful and then frozen. As models improve, the task set saturates and contamination risk rises.
Single-pass LLM synthesis scales the catalog, but difficulty often lives in the wording. Same-class models still solve those tasks, and correctness is left to LLM judges with position and verbosity bias. Environment generators, including Agent World Model, still pose the question first and only later check for a valid, checkable solution path. AutoGym, from Amazon AGI, reverses the order: specify the legal solution space, then instantiate the world.
The input is a domain seed (platform setting plus tool schema) or prior failure traces. AutoGym first writes a blueprint: which entities and relations make a legal solution, what the environment must contain, and which constraints the verifier will apply. A reconciliation pass checks that withheld facts are still discoverable and that the tool interface can support the intended path. Only then is the maze materialized. Solvability and verifiability are construction requirements.
Difficulty controls sit apart from the domain, so the same tools can host shallow lookup or cross-source reconciliation:
Ground truth is computed by running the blueprint derivation on the materialized tables, not by trusting numbers proposed at generation time. Deterministic outcomes are checked in code. Open-ended pieces go through a process judge, an outcome judge, and a meta-judge that emits a reward in [0, 1]. The curriculum reweights any dimension below 0.30 success (relax) or above 0.80 (tighten), with bounded updates so the distribution does not oscillate.
A procurement task shows the mechanism. The question only asks whether APAC vendors are hitting the mark this quarter. No field names, no threshold. The environment holds 800 delivery-log rows and 200 policy rows. Walking the solution path yields an on-time rate of 81.25% against a 90% SLA, so the vendors fail. Distractors include mid-quarter region reclassification, a retired commodity label, a competing 85% fill-rate policy, and 50-row pages while 96 records are relevant.
Generation uses Claude Sonnet 4.6. Agents under test see only the user task and the materialized environment. Eight independent rollouts per task. Bands follow mean verifier score: hard below 0.30, medium 0.30 to 0.80, easy at or above 0.80.
Productivity gym, same seven-tool interface:
| Suite | Opus 4.6 hard | gpt-oss-120b hard |
| Human expert, N=300 | 21.5% | 40.0% |
| Agent World Model, N=217 | 4.0% | 9.0% |
| AutoGym C1 (easier), N=297 | 10.0% | 33.4% |
| AutoGym C2 (harder), N=280 | 39.0% | 70.0% |
C1 and C2 differ only in topology weights, obfuscation, and distractor budget. C2 moves Opus hard share from 21.5% on the human suite to 39%; Agent World Model sits at 90% easy. On gpt-oss-120b, C2 is 70% hard, while Opus still places 61% of tasks in medium or easy. The split tracks capability. It does not look like uniform noise.
Repair rewrites how hard the gym looks. Before repair, 57% of 350 C2 candidates looked hard. After three iterations, 280 remain: hard falls to 39%, easy rises from 30% to 51%. A large slice of apparent hardness was a broken environment or verifier. Yield is 84.9% for C1 and 80.0% for C2.
The second setting starts from gpt-oss-120b failure traces, targets relative dates, time zones, and schedule overlap, and keeps 92 of 100 temporal-reasoning tasks. Opus is 89.2% easy; the weaker model has 41.4% in hard or medium. All 10 tasks outside Opus's easy band are hard for gpt-oss-120b. None reverse that order.
On a seven-dimension 0-100 audit, the productivity gym scores 92.7 on scope compliance and 91.4 on groundedness; interface realism is weakest at 78.0. Appendix F reports a 600-task, six-domain suite in under ten hours. A 50-task batch at 20-way parallelism costs about $100 to $200 and takes 54 minutes. Expert authoring is estimated at $500 to $1000 and 1 to 1.5 hours per task. An 8B model trained with GRPO for 500 steps on a 600-task instruction-following gym shows a rising reward curve. That is the only RL run, and the paper does not report start or end reward values.
The bottleneck for agent RL has moved to verifiable environments. AutoGym is a way to steer difficulty and grow tasks from observed weaknesses, at roughly two orders of magnitude below their estimated expert cost. It fits internal productivity tools, spreadsheet and document agents, and packing one capability axis against the edge of a model's comfort zone.
The increment is real and bounded. The paper credits Agent World Model as the prototype. What changed is making a legal solution a construction constraint and exposing knobs. There is no head-to-head on post-training gains against SWE-Gym or WebRL, and GRPO stops at one 8B model and 500 steps. This is a NeurIPS 2026 workshop paper. Treat it as an eval and data factory, not as evidence that the next OSWorld can be generated automatically.
Four caveats from the paper. Interface realism is 78.0 because tools are mocked, each task gets its own database, and schemas are simplified. Environment realism is 86.0; distractors sit unnaturally in the world. Open-ended tasks still need LLM semantic judgment. The cloud-emulator substrate is shown to materialize, not measured for difficulty or model separation. RL is one 8B run over 500 GRPO steps.
A few gaps sit outside that list. Certification scores are themselves LLM-judge numbers, which sits awkwardly next to the case against post-hoc judges. AWM kept 217 of 300 tasks after human review; AutoGym keeps tasks via automated repair, so the comparison is not fully symmetric. The temporal suite is already 89% easy for Opus: the curriculum is aimed at the weaker model's gap, not at keeping the frontier under pressure. High groundedness partly reflects an internal corpus and web retrieval that outside groups will not have.