NVIDIA paper: verify-then-execute lifts TerminalBench Pass@1 from 50.0% to 68.0%
dair_ai · x · 2026-10-04
An NVIDIA paper on test-time compute for terminal agents finds you should sample several candidate shell commands and verify them before running one — and spend more on the verifier than on extra samples.
Key results:
- With a GPT-5.6 Sol verifier choosing among 8 sampled actions, TerminalBench-Lite Pass@1 rises from 50.0% to 68.0%
- With a weak verifier, extra samples add almost nothing
Mid-Harness leaves the generator and harness unchanged and works between them. When the small TMAX-9B model verifies its own candidates, pairwise comparison works best, and distilling the strong verifier in helps further. Combining action sampling with trajectory sampling reaches higher success at lower token cost than sampling full trajectories alone.
More from coding & agent
- TanStack AI adds experimental text-to-GLB 3D model and prompt-to-motion animation generation — bennash · 2026-11-20
- Dev builds typing survival game with troops and skill trees in ~24 hours using agents and kaigen3d — hullabaloo22 · 2026-11-20
- Harmonic's Aristotle writes proven-correct fusion plasma controllers in C and Verilog in 4 hours — satnam6502 · 2026-11-20
- R3 Code: A Native Rust Mac Client That Runs All Your Coding Agents — jasonkneen · 2026-11-20
- zlib: 2.4K-star open-source CLI turns Z-Library into a skill for AI agents — udmrzn · 2026-11-20
- Andriy Burkov's True Positive Weekly #186: what to know when agents do the work — burkov · 2026-11-20