AutoEnvScaling: Automating the Data Flywheel with Terminal Agents
Simon Yu, Baolin Peng, Bo Liu, Zhengyan Shi, Zhaoyang Wang, Hao Cheng, Qianhui Wu, Wenlin Yao, Natasha Jaques, Weiyan Shi, Jianfeng Gao
cs.SE, cs.CL, cs.LG
2026-10-10
AutoEnvScaling casts RL environment design as a terminal task: valid-env rates rise up to 25x, and Qwen3.6 training on environments it designed itself gains 12.2 points on Terminal-Bench 2.1.
Terminal agents, which get work done by running commands in a shell, are trained with RL on large pools of diverse, verifiable task environments. Those environments are still mostly hand-built by researchers, and the paper cites reporting that companies are weighing annual spending above $1 billion on RL environments. Environments also depreciate: once the agent has learned what an environment teaches, someone has to re-diagnose its weak spots and write new ones. Supply grows at the pace of human labor.
Existing automation shares one blind spot. Pipeline synthesis fixes the generation recipe (domain taxonomies, skill lists, fixed stages) before training, so the pool never adapts; feedback-driven methods execute and repair generated tasks, but how the feedback is used is still hand-designed. Neither lets the process of making environments evolve with the solver.
AutoEnvScaling turns environment design itself into a terminal task and hands the researcher's whole workflow to a proposer agent.
Each round, the host reads the solver's latest rollouts, picks a parent environment, and gives the proposer an assignment with its pass rate and most common failure. If the solver keeps failing, the proposer builds an easier variant; if it passes reliably, the proposer lengthens the horizon or stacks on another skill.
The proposer works inside a sandbox with a coding harness. Its workspace holds the solver's trajectories and verifier outputs, previously generated environments, and memory and tools that persist and can be edited across rounds; it has web access, with egress rules blocking Terminal-Bench and other benchmark pages. It decides what to build and when the task is done: writes the files of a complete Harbor task (instruction, container environment, reference solution, tests), launches child containers to build the image and run the solution and tests, and revises until the tests behave. It can even run the current solver on a draft to estimate difficulty.
Two gates run outside the sandbox where the proposer cannot touch them. Validation requires the reference solution to earn full reward and an empty solution to earn less than half, plus a 13-gram contamination check against held-out tasks. Calibration requires the current solver's mean reward over several attempts to fall between 0.25 and 0.75 with variance across attempts, which keeps tasks at the edge of capability. Rewards stay simple: the solver earns the fraction of tests passed, the proposer earns -1, -0.25, or +1 depending on which gates its environment clears. Closed-weight proposers keep their weights fixed and improve only their harness across rounds; an open-weight model can play both roles, trained jointly with DPPO and with separate advantage baselines per role.
The load-bearing choice is making proposing a terminal task at all. Building an environment is writing and debugging code, the same skill distribution as solving, so what the model learns as a solver carries back into designing environments. That transfer is what makes the self-play experiment work.
Across eight frontier models, using DeepSeek-V4-Flash (the largest gain):
| Method | Valid-env rate |
| Prompt | 3.5% |
| Prompt + env feedback | 12.1% |
| AutoEnvScaling | 89.0% |
AutoEnvScaling wins on all eight models, up to 25x over prompt-only generation, and cuts cost per valid environment for six of eight, by up to 75%. Its tasks are richer: average solver turns rise from 33.8 to 47.2 while the solver pass rate falls from 61.0% to 51.0%.
RL gains (held-out Terminal-Bench 2.1, avg@5, matched training steps; Tmax is a static pool generated by the same proposer):
| Solver | Proposer | Tmax | AutoEnvScaling |
| Qwen3.5-9B | GPT-5.6-sol | 27.2 | 33.5 |
| Qwen3.5-9B | Claude Opus 5 | 29.2 | 36.6 |
| Qwen3.5-35B-A3B | GPT-5.6-sol | 34.6 | 38.9 |
| Qwen3.5-35B-A3B | Claude Opus 5 | 35.5 | 39.3 |
Removing the solver trajectories from the proposer's workspace costs the 35B model 3.6 to 6.7 points, roughly back to Tmax, so the gains track the solver-guided curriculum.
Self-play: Qwen3.6-35B-A3B plays both roles. The base model produces valid environments only 32.3% of the time; fine-tuning on 600 high-quality proposer trajectories (Cold-Start) raises that to 86.8%. It then generates environments, trains on them, and uses the updated checkpoint for the next round. Terminal-Bench 2.1 goes from 41.1 to 53.3 (+12.2, 6.1 above Tmax), and Terminal-Bench 4.0 from 1.6 to 3.2 even though the curriculum never targeted 4.0. Remove proposer-side training and the run collapses. Useful rollout groups run at 3.1x the Tmax rate and rollout collection takes 42% less time per step.
Two more studies round it out. On learning when to ask for help, RL on 500 generated under-specified software-engineering tasks lifts Ask-F1 from 29.6% (official training set) to 55.3% and task success from 6.8% to 33.9%, where frontier models top out at 14.4% and 7.5%. And the pipeline generates environments across seven domains (software engineering, computer use, GPU kernels, and others) with 71.3% acceptance over 1,489 runs, though no downstream training gains are shown there.
The bottleneck for agent post-training is shifting from compute to verifiable environment supply, and this paper shows the curriculum can run automatically at the frontier of the solver's ability and refresh as the solver improves. For practitioners, the concrete numbers are the 3.1x useful-group rate and 42% faster rollout collection: RL budget that would otherwise burn on zero-signal groups. Closed-weight models can serve as proposers through harness optimization alone, no weight access needed, so that path is copyable today. The self-play run is a small, reproducible demonstration of recursive self-improvement with no stronger teacher in the loop.
The authors list four: gains are not shown to persist past the evaluated runs; Terminal-Bench 4.0 sits at 3.2, so long-horizon tasks remain unsolved; passing validation does not make an environment correct, since the reference-solution and empty-solution checks cover a small slice of behavior and tests can miss bad solutions or reject good ones; and the cross-domain results show generation feasibility, not training gains. A few more caveats from the numbers. Cost per valid environment actually rises for the two strongest proposers (Claude Opus 5 at $1.18 and GPT-6-astra at $2.75, versus $0.72 and $1.79 for prompt-only), so the 75% saving is the weakest-model case. The self-play loop does not start from zero: the 600 cold-start trajectories and the admission gates are external inputs. The harbor check review is run by the proposer's own model, so author and reviewer share blind spots. And the valid-environment metric only checks that a task builds, the reference passes, and an empty solution fails, with no direct measure of task quality or diversity.