Software-in-the-loop: self-supervised scaling of terminal environments lifts Terminal-Bench 2 to 53.56%

Zhongzhi Li · hf · 2026-10-06

A paper introduces software-in-the-loop reconstruction (SWR), a self-supervised framework for scaling terminal-agent training environments into scientific domains:

Motivation: terminal agents are deployed beyond software engineering, but hand-authoring executable reference behavior and domain verifiers per task is costly and hard to reuse.
Method: derive reference outputs and verification targets from existing software workflows (executable input→output programs); run multiple input configurations, split into public observations and hidden evaluations; an agent builds a candidate program from instruction, schema and public I/O only, evaluated on hidden configs against workflow outputs; a hierarchical verifier combines domain-specific semantic comparison, structural validity and anti-shortcut checks, with public feedback enabling iterative revision.
Scale & results: 500 workflows across 46 software families in six domains; Qwen3.8-Max solved 838 tasks in three attempts, yielding 1,422 verified trajectories oversampled into 3,000 reconstruction-only training examples; SFT of Qwen3.8-27B lifts mean Terminal-Bench 2 from 47.94% to 53.56%, best among four matched-token corpus controls on all four reported evals.

Takeaway: existing scientific software alone can provide scalable, behaviorally verified supervision for terminal agents.

Original post →

Software-in-the-loop: self-supervised scaling of terminal environments lifts Terminal-Bench 2 to 53.56%

More from coding & agent