T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang
cs.LG, cs.AI
2026-09-10
Hunyuan post-trains Qwen3.5-122B-A10B with per-assertion execution rewards, lifting Terminal-Bench 2.1 from 43.8% base and 49.4% SFT to 64.0%, ahead of GPT-5.4 on the same harness.
The Linux terminal is a harsh test for long-horizon agents. The model issues real shell commands whose side effects persist, and a held-out verifier grades the machine state by execution, not a preference model. A single task can mean bisecting hundreds of commits, patching a defect, rebuilding, and proving the fix, over hundreds of turns.
Two failure modes show up at 122B MoE scale. Expert weights hold 116.0B of 121.4B parameters; each layer picks 8 of 256 experts. Tiny numeric differences between the inference stack and the trainer swap experts, so the gradient can hit a sub-network that never produced the sample. Multi-turn harnesses also re-decode and re-encode history at every turn boundary, so token IDs drift. Reward is the other bottleneck: a rollout batch costs hundreds of sandbox-hours, and a binary outcome returns one bit. An early binary-reward run on TMax-15k moved the Qwen3.5-122B-A10B base from 43.8% to 47.2% and never caught the 49.4% SFT checkpoint.
T1 asks whether MoE agent RL on executed outcomes can be made stable at this scale, and whether the gains transfer off the eval set.
Starting weights are Qwen3.5-122B-A10B. PPO runs on slime, with inference replicas and the trainer on disjoint accelerators. The agent drives a real shell in a cloud sandbox, up to 300+ tool-call turns in training. Evaluation uses Terminus-2 on Harbor, capped at 60 turns.
Tasks come from recursive synthesis (RST). Each round lengthens a seed's executable solution, then realigns the environment, verifier, and public instruction; the reference solution must pass a fresh sandbox. T1-15k keeps 15,000 of 37,484 synthesized tasks after an eight-dimension audit (verifier 45%, solution 25%, instruction 20%, task value 10%). Hidden requirements, test leakage, shortcut solutions, and weak verifiers are hard-rejected. Verifiers report per-assertion outcomes. The training pool is disjoint from Terminal-Bench 2.1.
Stability splits into two independent constraints.
TITO (token-in, token-out) feeds the trainer the token IDs the sampler actually emitted, so decode-then-encode cannot rewrite the condition. Observations and glue tokens enter as masked context. Turn boundaries are stitched under a small, auditable case set. On 1,402 production samples, drift inside the loss region is 0.0000%.
R3 (rollout routing replay) records the experts chosen at every MoE layer during generation and reuses that mask in the training forward pass. Softmax still runs on live logits, so the router remains trainable. Missing routing aborts the rollout rather than silently dropping the sample.
Together they cut the train-inference log-probability gap from 0.021 to 0.013. KL penalties are off: a frozen reference routes with its own selection and would charge the policy for bookkeeping. The MoE load-balancing coefficient is zero, because balancing fights replay.
Reward is passing-assertion count over a fixed global scale S=20, not a pass ratio. Ten of twenty on a hard task scores 0.5; two of four on an easy task scores 0.1. S sits near the 90th percentile of assertion counts in T1-15k (median 4, max about 35) and never changes, so the critic's regression target stays comparable across steps. The scalar lands on the last response token; GAE with γ=λ=1 carries credit back. One trajectory per task per step, so there is no group baseline. The critic is the only baseline.
That critic is a full-size second copy of the same architecture, warmed up for one epoch on TMax-15k. A cold start opened at explained variance -33.6 and stayed negative for 30 of 58 logged steps. After warm-up it sits between 0.71 and 0.86 from the first update. Critic LR is 1.5×10⁻⁵ against 1.0×10⁻⁶ for the actor. GRPO under the same budget scored 51.7% at both step 10 and step 20, with no reward trend. On long-horizon tasks a group that all fails (or all succeeds) contributes zero gradient; PPO buys the baseline that GRPO would have to purchase with repeated rollouts.
All numbers below use Harbor / Terminus-2. Terminal-Bench 2.1 is 89 execution-graded tasks.
| Model | TB 2.1 | LHTB |
| Qwen3.5-122B base | 43.8% | 18.9 |
| RST-SFT | 49.4% | 23.6 |
| Binary reward / TMax-15k | 47.2% | 20.3 |
| Dense reward / RST-38k | 59.9% | 25.4 |
| T1 | 64.0% | 27.9 |
| GPT-5.4 (same harness) | 54.8% | 27.2 |
| Claude Opus 4.6 | 63.8% | – |
| Claude Opus 4.7 | 66.1% | – |
Base to final is +20.2 points. SFT adds 5.6; RL adds 14.6, about three quarters of the climb. Under the same harness T1 beats GPT-5.4 and DeepSeek-V4-Flash at 56.9%, and sits 2.1 points behind Claude Opus 4.7, with 10B active parameters. GPT-5.4 on Codex CLI reaches 77.3%. Harness choice moves the score, and T1 was RL-trained for this one.
Long-Horizon Terminal Bench reports average reward, not resolved rate. T1 hits 27.9, matching Gemini-3.1-Pro, above GPT-5.4 at 27.2 and GLM-5.1 at 26.7. Claude Sonnet 4.6 scores below T1 on TB 2.1 yet 37.3 on LHTB, so the two rankings do not substitute. Terminal-Bench Hard: T1 38.0%, SFT 28.3%, base 20.0%, ahead of DeepSeek-V4-Pro at 36.0%.
Debugging (5 tasks) is 100.0%; system administration (9 tasks) is 88.9%, against 80.0% and 55.6% for GPT-5.6 Sol. The subsets are small. Easy is 100% at every checkpoint. Medium is 78 / 58 / 56%. Hard is 33 / 30 / 20%, so most of the RL-over-SFT gap sits on Medium. Evaluation turns average 94.4 for T1 versus 31.5 SFT and 41.1 base. On six unsolved tasks T1 ran 164 to 473 turns and timed out; the comparison model stopped in 6 to 40. During training, turns rose from 10.4 to about 20.9 and flattened.
The reusable piece is the recipe. TITO and R3 close the two bookkeeping holes that make sparse multi-turn RL lie to the optimizer. Dense execution reward makes partial trajectories learnable. Critic warm-up gives PPO a usable advantage on step one. For anyone training agents against real environments, those three matter more than the 64.0% headline.
The specialty is also the limit. The top five T1-15k categories (scripting, software development, sysadmin, environment setup, version control) are 67.9% of the pool. Data science is 3.7%, debugging 1.3%, performance 1.0%. File operations, scientific computing, and math still lag on the eval. This is terminal-agent post-training, not a general coding model.
TITO is exact only on the loss region; 2.6% of tokens in the audited run still condition later turns on harness re-tokenization. There is no single-axis ablation of binary versus dense reward: those campaigns also changed the pool, initialization, and routing replay. The verifier runs inside an agent-controlled sandbox with no runtime tamper checks; defense is the audit. Oversampling drops the slowest trials, and remaining eval failures concentrate on exactly those long tasks. The data mix is command-line heavy.
Eval settings are uneven too. Against GPT-5.6 Sol, T1 used 56k input versus 120k, 8,192 output versus 32,768, and thinking off. 64.0% is the score of a model RL-trained on this harness. That is not leaderboard fraud. Cross-model rank still needs a discount.