Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Minki Kang, Ryo Hachiuma, Shaokun Zhang, Subhashree Radhakrishnan, Yonggan Fu, Jindong Jiang, Mingjie Liu, Ehsan Hosseini-Asl, Yi Dong, Yu-Chiang Frank Wang, Byung-Kwan Lee
cs.CL, cs.AI, cs.MA
2026-09-30
NVIDIA's Mid-Harness verifies 8 sampled actions before execution in terminal agents, raising Pass@1 from 50% to 68% on TerminalBench-Lite and stacking with trajectory-level scaling.
Terminal agents (LLM agents that get work done by executing shell commands) act one stochastic sample at a time. Whatever the model emits gets executed, and every later decision sits on top of the environment that command produced. One bad action, an install of the wrong package or an edit to the wrong file, can make the task unrecoverable even though the same model could have sampled something better. The paper frames this as action reliability.
Existing test-time compute for agents is spent at the trajectory level: run T complete trajectories and pick one (Best-of-T), or run, summarize, and rerun (Sequential Refine). Both consume fresh environment runs, and cheap state snapshotting is not something real environments guarantee. The action level, sampling several candidates per step and picking one before execution, needs only one environment instance, but nobody had systematically mapped when it pays off. This NVIDIA and KAIST paper runs that study.
Mid-Harness is deliberately thin: it lives inside the model-call wrapper. The harness loop stays untouched; instead of one generate call per step, the wrapper draws N candidates from the same interaction history, runs a verifier over them, and returns one action. Generator weights and harness never change.
Three verification mechanisms are compared:
The verifier sees the task, the history, and the candidates, but not the reasoning the generator produced alongside them. Under self-verification the verifier is the same model as the generator. A LoRA-distilled variant trains that same backbone on 117k pairwise responses from GPT-5.6 Sol (732 trajectories, 244 hard tasks), still leaving the generator alone. A decision-only variant emits a single A/B token per comparison and scores candidates by token log-probabilities, skipping written rationales.
Main setup: TMAX-9B generator, TerminalBench-Lite (98 tasks), three runs per task, Pass@1 reported.
| Configuration | Pass@1 | Note |
| Base agent (N=1) | 50.00% | reference point |
| Zero-shot listwise, N=4 to 8 | 49.32% to 51.02% | wider sampling barely helps under weak verification |
| Zero-shot pairwise, N=8 | 54.76% | best zero-shot mechanism |
| Distilled pairwise, N=8 | 57.14% | +2.38 over zero-shot pairwise |
| GPT-5.6 Sol verifier, N=8 | 68.03% | upper-bound probe |
Same generator, same candidate pool: swapping in a frontier verifier moves Pass@1 from 50% to 68%, so the pool already contains good actions. Under weak verification, doubling width from 4 to 8 buys 1.7 points. The bottleneck is verification, not sampling. Distillation raises pairwise agreement with the teacher from 59.01% to 74.58%, yet still leaves an 11-point gap to the frontier verifier.
Verification is not cheap: at N=8, zero-shot pairwise costs about 125.7k verifier output tokens per run versus 0.6k for listwise. Decision-only verification at 9B raises Pass@1 while cutting token cost 20.9-24.1%; at 4B and 27B the savings come with a drop in Pass@1.
Composition with trajectory scaling: Best-of-3 alone reaches 55.10%; adding distilled Mid-Harness hits 66.33% on the same three environment runs, and one round of SR plus Mid-Harness hits 60.20%. Both beat Best-of-7 (59.18%) at lower estimated token cost, with the SR combination under half of it. Standalone distilled Mid-Harness at N=8 matches Best-of-5 (both 57.14%) at roughly a third of the cost.
Transfer holds across 4B (38.78% to 43.88%), 27B (71.09% to 76.19%), Nemotron3.5 Lightning, Nemotron3 Ultra 550B, and Qwen3.5-9B, plus Terminal-Bench 2.1, SWE-bench Verified Mini, FeatureBench-Mini, and the Terminus-2 harness. In all seven transfer settings, zero-shot Mid-Harness improves Pass@3 and matches or improves Pass@1.
For anyone serving agents, the practical claim is that action-level sampling plus verification is a separate, stackable axis of test-time compute, and the one that keeps working when the environment cannot be cheaply snapshotted or replayed. The implementation is a wrapper: no model change, no harness change, low engineering friction.
The second lesson generalizes further: sampling width is not where returns come from, verification quality is. This re-establishes for agents what the reasoning literature found about repeated sampling needing a verifier to pay off. If Best-of-T is already in your stack, raising per-trajectory quality beats raising T at these price points.
To be fair about novelty: sample-then-verify is not new (LLM-as-a-Verifier and action-ranking critics are close relatives). The value here is the controlled comparison and the mechanism analysis, an incremental contribution.
Stated by the authors: distillation still sits about 11 points below frontier verification, and the paper does not show how to close the gap; there are no gold action labels, so verification correctness and candidate coverage are measured only indirectly through trajectory success; every step gets full sampling and verification, with adaptive allocation left open; and there is no comparison against spending the same compute on longer reasoning for a single candidate.
Points that deserve scrutiny beyond that. The 68.03% figure is an opportunity probe with GPT-5.6 Sol, not a deployable number; the deployable distilled variant lands at 57.14%, a 7-point gain. Main results concentrate on one generator-benchmark pair: on Terminal-Bench 2.1 distillation does not beat zero-shot, and on FeatureBench-Mini it raises Pass@1 but lowers Pass@3 relative to zero-shot, so transfer is real but not monotone. TerminalBench-Lite has 98 tasks, so one Pass@1 point is roughly one task, and two-to-four-point gaps between mechanisms warrant caution (confidence intervals live in the appendix). Decision-only verification is free at 9B and costly at 4B/27B; the paper reports the pattern without explaining it.