10,159 real repos become training environments; a 35B agent overtakes GPT-5.5 on full-repo generation

From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop

Yang Tian, Tianyu Zheng, Xiyuan Yang, Yaxin Du, Chengran Yang, Bo Zhao, Jian Yang, Bryan Dai

cs.CL

2026-11-16

RepoFactory turns real repos into 10,159 verifiable whole-rebuild environments evolved by static code metrics, no rollouts; SFT on Qwen3.5-35B-A3B adds 22.6 points on NL2Repo.

What problem this solves

Training coding agents for long-horizon work is bottlenecked less by algorithms than by environments: tasks need executable verification, real length, and actual training value. Current pipelines fail in one of two ways. Linear builders such as DeNovoSWE and MindForge repair documentation and builds, then discard any repository that still fails validation, so scaling means hunting for more naturally compliant repos. Approaches that use model performance as feedback (BenchEvolver, SETA, DataMaster) must run a solver rollout, or even a training run, on every candidate before revising it; on long-horizon coding tasks a single rollout is itself a big job, so feedback is expensive and slow. The question the paper poses: can cheap, timely signals both rescue unsuitable repositories and raise the training value of the tasks themselves?

Method

RepoFactory converts a real repository into an environment (I, W, V, O): a language specification, an isolated workspace, a verifier, and the original code kept as the oracle. The task is full-repository generation: the agent gets only the spec and a dependency-installed, network-disabled empty workspace, and must write the entire project from scratch, scored by the repository's own tests.

Construction runs in four steps. Filtering keeps non-fork, unarchived repos with at least 10 stars, 5+ implementation files, under 40k lines of Python, and offline-runnable behavior; repos already in major benchmarks are excluded to avoid contamination. A Docker image is built per repo and kept only if all tests pass in a network-free container. A read-only AST scan extracts public APIs and the symbols tests import. A synthesis agent writes the spec plus an API manifest, checked against every symbol the tests import, because a functionally correct rebuild still fails if interface names differ.

The core piece is the Environment Evolution Loop, which maximizes a product Q = A · C · S:

The loop runs at most 12 rounds: secure A, fix whichever of S (threshold 0.5) or C (threshold 0.01) is below its bar, then push either term; every revision is revalidated, and a revision that does not strictly raise the targeted score is rolled back. Feedback comes only from verifier execution and static code analysis, with no solver rollouts and no downstream training. All construction agents use Qwen3.8-Max.

Results

The pipeline produced 10,159 environments across 17 software domains, the largest domain at 12.5%. The median oracle has 4.2k lines of code and 26 source files; verifiers total about 2.09 million test definitions. SFT data comes from running Qwen3.8-Max five times per environment and filtering low-reward trajectories: 10,947 trajectories over 5,577 environments, 64 assistant turns at the median.

Main results, all under the same Claude Code harness:

ModelNL2RepoSWE-bench Verified5-benchmark avg
Qwen3.5-35B-A3B (base)23.865.845.1
RepoHorizen (no evolution)41.370.951.5
RepoHorizen-35B-A3B46.472.253.5
GPT-5.545.182.667.6

The 46.4 on NL2Repo beats GPT-5.5 (45.1), Gemini-3.1 Pro (33.4), DeepSeek-V4-Pro (38.5), and MiniMax-M3 (42.1), the best listed score among open-weight models under 200B; Claude Opus 5 (75.3) and GPT-5.6-sol (56.8) remain well ahead. The other four benchmarks gain 1.3 to 6.8 points, and the evolution loop itself contributes +5.1 on NL2Repo and +2.1 on average over the no-evolution variant.

Two controlled comparisons carry more weight. Data source: same base model, same harness, about 11k trajectories each, RepoFactory reaches 46.4 while the strongest of six open SWE datasets, OpenSWE, reaches 25.2, with RepoFactory ahead on all five benchmarks. Environment scaling: performance climbs from 23.8 to 46.4 as environments grow from 0 to 5,577 with no saturation, and the curve steepens past 3,000, the last 2,577 environments lifting 36.5 to 46.4. Across 9B, 27B, and 35B-A3B model sizes the gains are 12.2, 14.1, and 22.6 points.

Mechanism checks: on 500 sampled repositories, evolution raises the yield of valid environments from 33.8% to 69.4%, mostly from the first repair; teacher success falls from 23.0% in the smallest-scope quintile to 13.4% in the largest, supporting S as a difficulty proxy; fine-tuning on 2,000 trajectories each, optimizing S lifts NL2Repo from 30.2% to 32.9%, and optimizing C reaches 32.5% with fluctuation in between.

Why it matters

For teams building agent training data, the most transferable finding is that environment synthesis does not need expensive rollout feedback: a product of static code metrics is enough to drive evolution, and doubling the yield means twice the tasks from the same repository pool. Full-repo generation also fills a gap in task formats: the SWE-bench family tests fixing issues inside an existing repo, while building a multi-file project from scratch, with components that have to interlock, previously lacked verifiable environments at scale. To be honest about it, each ingredient here (filtering, containerization, spec synthesis, the loop) is known engineering; what is new is the product objective and the closed loop, a solid but incremental combination.

Limitations

The authors state two: Python only, and proxy metrics that describe coarse structure without distinguishing which capabilities a task actually exercises.

A few more caveats from the reading. The largest gain lands on NL2Repo-Bench, which matches the training task format (rebuild a repo from a language spec), so the 22.6 points are partly in-distribution; transfer to the SWE-bench family shrinks to 1.3-6.8 points. Only 5,577 of the 10,159 environments produced training samples, silently dropping the ones the teacher could not solve, so the hardest tasks never entered the training set. The S and C ablation gains are under 3 points and the C curve fluctuates, so the causal link between proxy metrics and training value is not iron-clad. Training is SFT only; the RL half of the verifiable-environment story has no experiments behind it yet. Table 2 also mixes numbers reported by prior work under varying harnesses with the paper's own controlled runs, so cite with care.

Terms

Source

Related papers

All paper explainers