MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis
Yihao Chen, Shi Chang, Khaled Chawa, Feng Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
cs.SE, cs.CL, cs.LG
2026-07-30
MindForge strips source from open-source CLIs into from-scratch coding environments, distills GLM-5.2 trajectories into Qwen3.6-27B, and lifts ProgramBench pass rate from 37.98% to 49.51% past Sonnet 4.6, with gains on seven unseen SE benchmarks.
Coding agents have gotten good at modifying existing codebases: fixing bugs, adding features, all with the source in front of them. Building a complete program from scratch is a different and harder setting. There is no implementation to extend, so the agent has to infer the full specification from documentation and the reference program's behavior, design an architecture, implement it, locate its own bugs, write tests, and iterate. ProgramBench tests exactly this, and even frontier models like GPT-5.5 fully resolve fewer than 1% of tasks.
The bottleneck is data, not the model. Existing environment-construction frameworks cover only one phase of development, such as bug fixing; none span the whole life cycle of from-scratch construction.
MindForge is an automated pipeline that turns open-source command-line programs into source-free environments, in two stages.
Stage one builds the environment, producing a cleanroom image that contains only the compiled reference executable and sanitized public documentation; the agent never sees source. Five steps: pick a repository at a pinned commit; an explorer agent screens it offline reading only source and docs; a builder agent discovers the build and writes a self-contained build script; a behavior-equivalence check rebuilds in a fresh sandbox to confirm identical behavior; a source-free check scans the binary to prevent source leakage. Going source-free is the point: it forces the agent to work like it inherited a black-box binary, with no source to peek at.
Stage two collects trajectories. GLM-5.2 acts as the teacher agent, running whole-life-cycle trajectories inside cleanroom environments, then two refinement passes clean them up: infrastructure-noise recovery rewinds and resumes from transient failures, and reasoning rewrite repairs orphaned reasoning left after tool-call errors are removed. Qwen3.6-27B is then fine-tuned on 973 trajectories averaging 181.6 turns and 177K tokens.
ProgramBench average pass rate: GLM-5.2 (teacher) 64.60%, GPT-5.5 56.50%, Claude Opus 4.7 51.38%, Sonnet 4.6 47.97%, Qwen3.6-27B base 37.98%, MindForge-27B 49.51%. Fine-tuning lifts the 27B from 37.98% to 49.51%, overtaking Sonnet 4.6 and approaching much larger frontier models.
It also generalizes. On seven unseen software-engineering benchmarks it beats the base everywhere: RepoZero-C2Rust +31.00 points (47% to 78%), DeepSWE +14.16 (1.76% to 15.92%), NL2Repo-Bench with/without tests +10.70/+4.56, SWE-bench Verified +5.04, SWE-bench Pro +5.93, SWE-bench Multilingual +5.22, FeatBench +4.94. These span long-horizon repo generation and translation, bug fixing, feature implementation, and cross-language issue resolution.
Behaviorally the fine-tuned model is markedly more patient: mean turns more than double (344 to 735.7), tool calls roughly double, yet the command-failure rate actually drops (10.98% to 9.35%), and the rate at which reasoning turns into real implementation edits nearly doubles.
This shows that the from-scratch building ability, the capability frontier models most dominate, can be distilled onto a 27B model with automatically built environments and a large teacher, and that the skill transfers across tasks. For teams that need localization, private deployment, or cost control, that is a cheaper path than scaling parameters. The method is reusable on its own: 562 source-free environments across six compiled languages (Go, Rust, C, C++, Swift, TS) are a scarce asset.
The authors concede that long-horizon benchmarks were run only once due to compute cost, so statistical robustness is limited, and that trajectory refinement may not catch every genuine error. Overlap with some benchmarks is tiny (DeepSWE: 2 repos; SWE-bench Multilingual: 3 repos), which they call negligible contamination, but that is a judgment, not a proof. There is also a ceiling: at 49.51% the student still trails GPT-5.5 (56.50%) by seven points and the teacher GLM-5.2 (64.60%) by fifteen, so the distillation itself has a hard upper bound.